What's Actually Inside a Production Voice Agent

The latency budget

A working voice agent has about 600 milliseconds to respond before the caller perceives awkwardness. That number isn't made up — it's the point where silence starts to feel like confusion rather than thought.

Here's where those milliseconds go:

• STT (speech-to-text): 150–250ms for streaming transcription. Deepgram and AssemblyAI return partial results fast, but you still wait for the end-of-utterance signal before deciding the user is done speaking.
• Intent routing + context retrieval: 50–150ms. If you're hitting a vector store or calling an API to check account status, this is where it happens. Keep your indexes warm.
• LLM inference: 200–400ms for first token with a fast provider (Anthropic, OpenAI). Streaming helps perceived latency, but your TTS engine can't start until it has a grammatically stable chunk — usually 8–15 tokens.
• TTS (text-to-speech): 100–200ms to first audio byte. ElevenLabs and PlayHT stream well; legacy engines don't.

Add 50ms for network hops and you're at 550–1,050ms. The difference between a tight system and a sluggish one is usually in the middle two: slow database queries, cold model endpoints, or naïve prompt chains that wait for full completion before acting.

We keep a Prometheus dashboard that tracks p50/p95/p99 for each stage. When p95 response time crosses 800ms, we know which component to optimise before users complain.

Retries and recordings

Production voice agents fail in ways demos never do. The user's connection drops. The LLM returns malformed JSON. The TTS service times out. Your job is to make these invisible.

Retries: We wrap external calls in retry logic with exponential backoff, but not all failures are equal. If STT fails mid-sentence, we apologise and ask the caller to repeat. If the LLM hangs, we fall back to a cached response tree — a decision-tree script that covers the most common intents. It's less flexible but fails predictably. If TTS fails, we queue the audio and keep talking from a buffer. The caller shouldn't know.

Recordings: Every call gets recorded and stored with metadata: transcript, intent log, latency percentiles, error events. We keep recordings for 90 days and transcripts for a year. This isn't compliance theatre — it's how we improve the system. When a conversation goes sideways, we replay it, find the brittle intent boundary, and add test cases.

We also flag calls where the agent said "I didn't catch that" more than twice. High confusion rate in a specific flow means the prompt or the training data is wrong. Fix it in the next deploy.

Graceful degradation

The agent needs an escape hatch. If it can't parse the user's request after two tries, it doesn't loop forever — it transfers to a human or offers to send an SMS with a callback link. We route this through the same telephony stack (Twilio, usually) so the handoff is clean.

We also build a "confidence threshold" into intent classification. If the LLM returns a score below 0.7, the agent doesn't guess — it asks a clarifying question or offers multiple options. Guessing wrong costs more trust than admitting uncertainty.

Why conversation flow is the smaller half

Most demos show a happy-path conversation: user asks a question, agent answers, done. In production, the conversation graph is the easy part. You map intents, write prompts, test a few variants, ship it.

The hard part is everything else:

• State management: Keeping track of what the user said three turns ago, which form fields are filled, whether they've been authenticated. We use a session store (Redis) with a 30-minute TTL and serialise state as JSON. Every turn writes to it; every prompt reads from it.
• Error instrumentation: Knowing when and why things break. We log every LLM call with input, output, latency, and token count. When GPT-4 starts refusing a prompt due to a policy change, we catch it in staging.
• Telephony quirks: Handling DTMF tones (some users press buttons mid-conversation), regional accents, background noise, crosstalk. Your STT model needs to be robust or you need fallback logic.
• Cost control: Voice agents burn tokens. A 10-minute call can easily hit 8,000 tokens with context window and retrieval. We cache repeated queries, prune old conversation history, and rate-limit expensive operations.

We've shipped agents that handle 2,000 calls a day. The conversation script was two weeks of work. The infrastructure, monitoring, and failure modes were two months.

What this means for you

If you're building a voice agent, start with the latency budget and work backwards. Measure every stage, not just the LLM. Design for failure — retries, fallbacks, handoffs. And don't spend all your time on the script. The conversation is the interface; the system underneath is the product.