On August 12, 2026 in Audio, Computing, IoT by SJ Barak

As large language models (LLMs) like ChatGPT become increasingly entrenched in people's lives, we like to fool ourselves into thinking they simplify things and create efficiency. Look closer, however, and we see something interesting happening behind the scenes, especially when it comes to voice queries.

Currently, when you use voice to ask artificial intelligence (AI) a question, several steps occur at the architectural level. First, voice activity detection occurs, followed by automatic speech recognition (ASR), which converts those words into text. Next, the LLM generates a text response, followed by text-to-speech (TTS) to read it back to you.

For now, it works and may even feel impressive, but having text as an intermediary is clunky, slow, and removes nuance. In this blog, we’ll examine the core bottleneck in today’s voice AI—the latency and loss of nuance introduced by text-based pipelines—and trace the shift toward native voice AI architectures.

From Text LLMs to Native Voice AI Models

Engineers may argue that only latency truly matters, but stripping speech of nuance creates significant limitations. When we speak, we aren't simply transmitting words. Humans convey emotion through signals such as pitch, timing, and sarcasm. Text flattens everything into a sequence of tokens easily processed by a traditional LLM, but it cannot detect an eye roll or gritted teeth, and that can change the meaning entirely. As a result, systems that convert speech to text lose important contextual information. Preserving the vocal signals that convey intent, emotion, and context requires a different approach—one that works directly with the audio itself rather than a textual transcription. Native voice AI models process and generate speech directly as audio, without requiring text as an intermediate stage. Instead of treating language as a sequence of written tokens, these models operate on compressed audio representations, allowing them to capture not just what is said, but how it’s said—intonation, timing, emotion, and subtle conversational cues.

The distinction is more than architectural. Traditional pipelines “strip-mine” speech for words and discard everything else. Native voice AI models preserve the full signal, enabling more natural interaction while also reducing latency by eliminating multiple processing stages.

Limitations of Removing Speech-to-Text

Why not cut out the speech-to-text part entirely, then? The reasons are severalfold. For starters, text is convenient as a medium for LLMs because it's already discrete. Every word can be represented as a token from a finite vocabulary, and predicting the next token is exactly what LLMs are trained to do.

Meanwhile, audio is continuous and data-intensive. A single second of speech at compact disc (CD) quality is 44,100 samples. Even at telephony-grade 8kHz, that works out to 8,000 samples per second. Training an LLM to predict the next audio sample would be computationally expensive, and models may struggle to learn meaningful patterns within such high-resolution data.

Neural Audio Codecs: Turning Sound into Tokens

A promising approach is neural audio codecs, which compress audio into a much smaller sequence of discrete tokens. SoundStream from Google and EnCodec from Meta pioneered this approach, using encoder-decoder neural networks combined with residual vector quantization (RVQ).

The process is clever. A neural encoder compresses the audio waveform into a latent representation, and a series of quantizers discretize that representation. In simple terms, the encoder produces a coarse representation, while the quantizers map it to a finite set of learned patterns. This allows continuous audio to be represented as a sequence of codebook indices.

Earlier quantizer stages generally capture coarser information, while later stages refine the residual and improve reconstruction fidelity. In some newer architectures, the token hierarchy is designed to better separate higher-level semantic information from finer acoustic detail. At the decoder end, these tokens are reconstructed into a waveform.

With neural codecs, a second of audio can be compressed into a much smaller stream of discrete tokens, though the exact token rate depends on the codec design and bandwidth setting. While audio still requires more tokens than text, which may represent the same content with only a few tokens, the resulting token stream is tractable for modern architectures. The audio tokens can be fed into an LLM architecture and used to train it to predict the next token in the sequence.

Real-Time Breakthroughs: GPT-4o and Moshi Native Voice AI Models

Recent systems demonstrate a massive architectural leap in native audio modeling. OpenAI’s GPT 4o is described as an end-to-end multimodal model across text, vision, and audio, capable of low-latency audio responses. Its real-time system can achieve round-trip latencies on the order of a few hundred milliseconds.

Kyutai, a French AI research lab, open-sourced a model called Moshi^[1] in September 2024 that further pushes this idea. Moshi handles both streams of a conversation—what you're saying and its own generated output—in parallel, enabling it to handle interruptions more naturally.^[2] There's no strict turn-taking protocol. It can process incoming audio while generating responses. Their measured latency is around 200 milliseconds, close to natural conversational pacing.

The underlying architecture is particularly important. Kyutai built a seven-billion-parameter transformer that predicts audio tokens from its neural codec, Mimi, and introduced an "Inner Monologue" mechanism that predicts aligned text tokens as a prefix for each audio token. This hybrid approach can improve coherence compared to pure audio-to-audio generation, suggesting that text may still serve a useful role as an internal representation even when it's not part of the input or output.

Google's AudioLM and SoundStorm take a different approach by separating "semantic" tokens, which represent content, from "acoustic" tokens, which represent how it is expressed, and generating them in stages. Semantic tokens are generated first to establish structure, followed by acoustic tokens that refine voice, timing, and recording characteristics.

Despite this progress, speech LLMs remain an evolving area. The models do not yet fully utilize all the information present in the audio signal. Part of the challenge is training data. Text models benefit from vast amounts of written content, whereas speech models require large volumes of paired conversational audio, which are harder to obtain at scale. Moshi, for instance, was initially pre-trained on 7 million hours of large-scale audio data accompanied by transcripts, post-trained on multiple audio streams, and fine-tuned on 170 hours of natural and scripted conversations between multiple pairs of participants and 2,000 hours of multi-channel telephone conversations.^[3] That's impressive for a research project, but it's tiny compared to the trillions of tokens used to train frontier text models.

Another challenge is latency sensitivity. Text chatbots can tolerate response delays of several seconds without significantly affecting usability. In voice interactions, however, delays above roughly 500 milliseconds are often perceived as disruptive.^[4] This constraint limits the size and computational complexity of models that can operate in real-time.

There's also ongoing discussion about whether transformers are the most appropriate architecture for audio. Transformers scale quadratically with sequence length because of how self-attention works: every token must attend to every other token. Audio sequences are long. A 30-second clip might be 3,000 tokens even with aggressive compression.

State-space models (SSMs), such as Mamba, offer an alternative by scaling linearly with sequence length and maintaining a fixed-size hidden state, improving efficiency for long sequences.

Hybrid architectures that combine transformer layers with SSM layers, such as Jamba from AI21, may offer a practical path forward. These approaches leverage transformers for global context through attention, while using SSMs for more efficient sequential processing over long time scales.

This distinction matters because voice is a primary mode of human communication, whereas text is largely a structured interface for machines. If AI systems can genuinely understand and generate speech natively while preserving nuance, that changes what kinds of interactions are possible.

For example, voice assistants could adapt responses based on vocal cues such as frustration, customer service systems could detect hesitation and respond more appropriately, and translation systems could preserve not just meaning but also tone and emotional context. These capabilities improve accessibility by reducing reliance on text-based interfaces.

The pipeline approach (ASR -> LLM -> TTS) will keep improving and will likely remain the practical choice for many applications for years to come. It's debuggable and allows individual components to be optimized independently, while enabling inspection through intermediate text representations.

The models skipping the middle layer entirely are improving steadily, and the threshold for "good enough" conversational performance may arrive sooner than expected, enabling more natural and fluid voice interactions.

Conclusion

Reducing voice AI latency is about both speed and restoring the richness of human communication that text pipelines remove. As native audio models mature, the key benefit will not only be faster responses but also more natural and expressive interactions.

[1]https://labs.scaleway.com/en/moshi/
[2]https://arxiv.org/html/2410.00037v2#S3.SS2
[3]https://arxiv.org/html/2410.00037v2#S4.SS1
[4]https://inworld.ai/resources/how-to-evaluate-tts-models; https://tringtring.ai/blog/technical-deep-dive/understanding-latency-in-ai-voice-agents-why-sub-500ms-matters/