Nvidia's new audio model does five jobs at once and beats the specialists at their own game

Nvidia released Audex, a unified audio-text large language model that folds speech recognition, translation, text-to-speech, audio generation, and speech-to-speech generation into a single transformer decoder. The model is built on the MoE backbone Nemotron-Cascade-2-30B-A3B: it activates only 3 billion of its 30 billion total parameters per token, which puts it in the same bracket as task-specific systems that often require multiple specialized models.

The architectural bet is on simplicity. Audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. Nvidia avoids complex modality-specific modules and lets Audex use standard LLM training and inference pipelines without custom infrastructure.

Training at scale

Nvidia built a training corpus of 157.4 billion audio tokens and 320.5 billion text tokens, covering speech, music, environmental sounds, and multi-speaker conversations. The pipeline applied multi-stage supervised learning followed by text-only Cascade RL and multi-domain on-policy distillation, a technique that uses a mix of reward signals from both text and audio tasks to align the model's outputs.

The distillation stage stands out. Instead of relying on human feedback or static benchmarks, Nvidia used on-policy sampling across multiple audio domains to generate synthetic preference data that mimics real-world use cases. This helped Audex preserve the backbone's performance on reasoning, alignment, knowledge recall, long-context understanding, and agentic tasks, areas where unified audio-text models often regress compared to their text-only counterparts.

Performance highlights

Benchmark results from Nvidia's research team show Audex beating older unified models such as Seed-ASR, Qwen2-Audio, and SpeechGPT on speech recognition (Word Error Rate on LibriSpeech and Common Voice), speech translation (BLEU on CoVoST-2), and text-to-speech naturalness (Mean Opinion Score on LJSpeech). On audio understanding tasks including sound event detection, music genre classification, and audio captioning, Audex sets new state-of-the-art scores on ESC-50, AudioSet, and Clotho.

The text-only Cascade RL stage was designed to prevent catastrophic forgetting. On standard NLP benchmarks like MMLU, GSM8K, and HumanEval, Audex shows marginal or no regression relative to the Nemotron-Cascade-2-30B-A3B backbone. That suggests Nvidia solved a long-standing tension in multimodal LLM training: adding audio understanding often degrades pure text reasoning, but here it does not.

Implications for the audio AI stack

Audex is being released as open model checkpoints, both the full 30B MoE variant and a smaller 2B dense version, which pushes the field toward a single-model approach for audio and speech tasks. Until now, production systems typically ran separate pipelines: one ASR model, one TTS model, one sound event classifier, and a text LLM for reasoning. Nvidia's unified architecture collapses that stack, reducing latency, maintenance burden, and hardware requirements.

The model also handles speech-to-speech generation, taking spoken input and producing spoken output directly, which opens use cases in real-time voice assistants and conversational agents without cascaded ASR-TTS loops. Early community feedback on Hugging Face highlights Audex's ability to handle code-switching, multi-speaker scenarios, and environmental audio prompts in a single inference pass.

Nvidia has not disclosed specific inference latency numbers or GPU requirements beyond noting compatibility with standard LLM inference frameworks. The checkpoints are available under the Nemotron research license, which permits non-commercial use and research reproduction.