How we made a text-to-speech model respond in sub-50 ms
Nari Labs published a blog post detailing their implementation of Qwen3-TTS 1.7B CustomVoice, which they claim achieves sub-50 ms p95 time-to-first-audio (TTFA) and 10 requests per second (RPS) on a single NVIDIA H100 SXM, while maintaining real-time playback. They define real-time TTS as requiring low audible TTFA, zero underruns, capacity as RPS increases, and non-malformed output. They compared five implementations—theirs, vLLM-Omni, SGLang-Omni, VoxServe, and M*—under Poisson open-loop traffic, tuning each for low-latency streaming. Theirs was the only one to achieve sub-50 ms p95 TTFA, and they maintain this latency through 10 RPS, keeping it below 100 ms even at 20 RPS. The system produces approximately 630 characters per second at 10 RPS. At $4.29 per hour for a 1x H100 SXM instance, they estimate a cost of about $2 per 1M characters at full utilization, comparing favorably to ElevenLabs V3 at $100 per 1M and Cartesia Sonic 3.5 at $49 per 1M, which have higher TTFA. They open-sourced the implementation and benchmark, and the post includes a methodology explanation.
Developers can now achieve sub-50 ms TTS latency at scale, potentially reducing costs for real-time voice applications.