Skip to content
ZJ

Case study /06 · Prototype · in development

Realtime Voice Agent

AI · Voice · Real-Time

A low-latency voice agent that holds natural phone conversations — booking, qualifying and answering without the robotic pauses.

The Realtime Voice Agent answers a phone call, understands what the caller wants, checks real systems mid-conversation — availability, records, bookings — and speaks its answers naturally, including when the caller talks over it.

  • Python
  • Telephony / WebRTC
  • Streaming STT
  • LLM streaming
  • TTS
  • VAD
  • Tool calling
  • Latency budgets
Realtime Voice Agent end-to-end conversational flow: an incoming call streams through speech recognition, LLM reasoning, speech synthesis and playback, with mid-call tools and an always-on turn-taking and barge-in controller that can interrupt any stage.

Context

Text agents get to hide latency behind a typing indicator. Voice gets nothing: dead air after a question reads as failure within a second. Making an LLM conversational over the phone is a systems problem disguised as an AI problem.

Problem

Three streams must overlap instead of queueing: transcription while the caller speaks, generation while the sentence completes, synthesis while later sentences are still being generated. Add interruptions — humans barge in constantly — and the pipeline must also cancel gracefully at any point.

Constraints

  • First audio in reply had to land within conversational rhythm — a hard budget, allocated per stage.
  • Barge-in: caller speech immediately halts playback and generation without losing conversation state.
  • Tool calls (lookups, bookings) had to happen mid-call without freezing the conversation.
  • Telephony audio is hostile: compression, noise and crosstalk were the baseline, not the edge case.

Responsibilities

  • Streaming pipeline architecture across STT, LLM and TTS stages.
  • Turn-taking and voice-activity detection tuned for real phone audio.
  • Barge-in cancellation semantics across all three stages.
  • Mid-call tool integration with conversational cover (acknowledgments while systems respond).
  • Per-stage latency instrumentation and budget enforcement.

Architecture

  • Audio streams into incremental STT; stabilized partial transcripts feed the LLM before the caller finishes speaking.
  • Generation streams sentence-by-sentence into TTS, so the agent starts speaking while still deciding how the answer ends.
  • A turn-taking controller arbitrates who holds the floor, driven by VAD and transcript stability rather than fixed silence timers.
  • Barge-in triggers cascade cancellation — playback stops, synthesis flushes, generation aborts — while the transcript records exactly where the agent was cut off.
  • Tools execute asynchronously; the agent bridges wait time with natural acknowledgments instead of silence.

Key decisions

  • Sentence-level pipelining over full-response synthesis: the single biggest perceived-latency win.
  • Budget latency per stage and instrument each one — 'it feels slow' becomes a graph, not a vibe.
  • Treat interruption as a first-class conversational move, not an error: the agent yields immediately and resumes context-aware.
  • Constrain the agent to its domain hard; a voice agent that rambles off-script erodes trust faster than one that transfers to a human.

Implementation

  • Partial-transcript stabilization avoids reacting to words the recognizer later rewrites.
  • The cancellation path is exercised in tests as heavily as the happy path — barge-in is constant in real calls.
  • Call summaries and structured outcomes (booking made, question answered, transfer) emit at hangup.
  • Human transfer hands off with a spoken context summary so callers never repeat themselves.

Performance

  • Overlapped streaming keeps first-audio response inside the conversational budget.
  • Stage-level latency traces expose regressions the moment any model or provider changes.
  • VAD tuning against real telephony noise cut false interruptions substantially.

Outcomes

  • Natural back-and-forth phone conversations with mid-call system lookups.
  • Interruption handling that callers stop noticing — which is the point.
  • A latency instrumentation rig that makes voice quality an engineering metric.
  • [PLACEHOLDER — add call volumes and latency percentiles when publishable.]

Lessons

  • In voice, latency architecture is the product; the model is a component.
  • Design the cancellation path first — real conversations are made of interruptions.
  • Phone audio humbles every demo; test against the network, not the microphone.

Software Architecture

Seven-stage flow covering answer, understand, act and respond in real time. 1. Incoming call: the caller connects over the phone network or WebRTC via PSTN, SIP or WebRTC, audio arriving as mu-law or Opus, starting a call session. 2. Streaming STT: speech to text incrementally, with voice activity detection, partial transcripts, and stability and confidence scoring, emitting a live transcript. 3. LLM reasoning and generation: understand, reason and generate a response using context and memory, tool planning when needed, and streaming tokens and sentences. 4. Streaming TTS: text to speech sentence by sentence with neural TTS, low-latency synthesis and audio frames streaming out. 5. Audio playback: audio out to the caller with natural pacing, using a jitter buffer, adaptive chunking and smooth playback. 6. Mid-call tools: lookups, bookings and system actions across availability and records, bookings and updates, executed asynchronously. 7. Call outcome: wrap up and deliver the outcome — summary, saved call artifacts, call ended or transferred. An always-on turn-taking and barge-in layer reaches into stages two through six: detect caller speech, interrupt playback, cancel TTS and LLM generation, preserve context, resume when the caller pauses. Four principles close the diagram: an end-to-end latency budget from first audio in to reply within conversational rhythm, overlap everything so listen, think and speak run at the same time, graceful cancellation that stops fast and keeps context, and human handoff with full context as a spoken summary.
Architecture diagram of a low-latency, interruptible, tool-using voice pipeline, lettered A to I. A, audio ingestion: telephony and WebRTC over PSTN and SIP, audio normalization for noise, AGC and codec, and voice activity detection for endpointing and turn taking. B, streaming speech to text: a streaming STT engine, a partial transcript stream that is incremental and token-level, and stability and confidence review. C, streaming LLM reasoning: context manager holding memory, history and user profile, intent and reasoning, a tool planner that decides the action, and a response generator streaming sentences. D, asynchronous tools and integrations reached by tool calls from the reasoning stage: accounts and CRM, availability and search, bookings and orders, payments and refunds, and a knowledge base. E, streaming text to speech: sentence buffer and pacing, a low-latency neural TTS engine, and a chunked audio frame stream. F, audio playback: jitter buffer, interruptible playback with smart barge-in, and network-aware adaptive output. G, outcomes and handoff: structured call summary, stored transcripts and audio, and human handoff with full context. Underneath, a turn-taking and barge-in controller acts as the system backbone, chaining detect caller speech, interrupt playback, cancel TTS, cancel LLM generation, keep context and resume when paused, with control arrows back into ingestion and outcomes. I, observability and data: latency traces, quality metrics, logs and events, analytics and alerts. A legend distinguishes audio, transcript, text and reasoning, speech synthesis, playback, control and interrupt, tool and data, and observability flows.