Free playbook
← All playbooks
InfrastructureWe run this

Run your customer agent on a model you own

Customer-facing replies come from an open-weight model on your own hardware, reached over a private network.

For
Businesses that don't want customer conversations sent to a public AI API
Setup time
1 day
Runs on
Local modelCustom code

Why we run it

Fernanda, our restaurant agent, answers from a Qwen model running on an NVIDIA GX10 in our office. The VPS that talks to Meta reaches it over a private Tailscale network.

Steps

  1. Pick a small, fast open-weight model. Customer FAQs need speed more than genius.
  2. Serve it with an OpenAI-compatible endpoint so your agent code doesn't care where it runs.
  3. Connect the public server to the model box over a private network. Never expose the model port to the internet.
  4. Allow only known endpoints in the agent's config and refuse to start otherwise.
  5. Keep a fixed fallback reply (and a human handoff) for when the model is slow or down.
The promptIn the kit
# agent config (example)
MODEL_BASE_URL=http://<private-address>:8000/v1   # OpenAI-compatible, private network only
MODEL_NAME=<your-open-weight-model>
Copy-paste kit · $29Full prompt and config, setup checklist, standing guardrails, updates.

Also in The Revenue Desk, The Full Bench.

Guardrails

  • The model port is reachable only on the private network.
  • The agent refuses to start with an unknown model URL.
  • A short timeout plus a fallback means a slow model never leaves a customer waiting.

What broke

  • Running a second large model and a big download on the same box at the same time starved the first one. We now keep one large runtime per machine.

Our stack

NVIDIA GX10, OpenAI-compatible server, Tailscale, Python agent on a VPS.

Want it running on your accounts?We install and adapt it for you, from $500.
Have us install it
Next playbookMake your website readable by AI assistants →