NVIDIA’s new speech model cuts real-time chat latency to under half a second
NVIDIA has pushed real-time spoken AI a step closer to human-like flow with NemotronLabs VoiceChat 11B, an open 11-billion-parameter model that handles streaming speech understanding and generation in a single network. The result is a system that listens while it talks, allows mid-utterance interruptions, and can call external tools on the fly—all with a measured turn-taking latency of 448 ms on the Full-Duplex-Bench 1.0 benchmark.
One network, no handoffs
Most voice agents chain automatic speech recognition, an LLM, and text-to-speech, creating latency and orchestration overhead. VoiceChat 11B collapses these steps into an end-to-end pipeline: a Fast Conformer encoder streams 16 kHz audio into a Nemotron Nano v2 LLM backbone, which simultaneously predicts text tokens and drives a TTS decoder that renders 22.05 kHz speech. The model’s hybrid Mamba/Transformer design, trained on roughly 550,000 hours of audio, allows it to start generating responses before the user finishes speaking and to yield instantly when interrupted.
Live tool calls without silence
Tool-calling is handled via a dedicated side channel that emits a <TOOLCALL> block while the main audio channel keeps flowing. Operators can pre-define “on-hold” lines that the agent speaks the moment a tool call is triggered, preventing dead air while APIs respond. NVIDIA cautions that the feature is constrained: no more than five tools per session, no simultaneous calls, and no user interruptions during tool execution. System prompts and tool responses must also be ASCII-only and TTS-friendly.
Ready for pilots, not production—for now
NVIDIA labels the release “research purposes only,” citing documented failure modes such as a two-minute audio context ceiling, drift into gibberish after several turns, runaway self-talk, and dropped words in user transcriptions. The weights and container are open and permissively licensed, but deployment today is limited to teams with an 80 GB+ GPU (A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux). There is no hosted API, so organizations without GPU access cannot evaluate the model directly.
Why it matters
VoiceChat 11B demonstrates that a single, unified model can rival cascaded stacks in responsiveness while adding live tool use—critical for contact centers, in-car assistants, and accessibility tools where speed and continuity matter. The open release lowers the barrier for experimentation, but the documented constraints and “research only” label remind us that production-grade, barge-in-capable agents remain a work in progress. Teams that can spin up an 80 GB GPU now have a sandbox to prototype next-generation voice experiences before commercial offerings mature.
Source: MarkTechPost. AI-assisted editorial synthesis — TechnoExpress.