What is Multi-Token Prediction That Speeds Up LLMs 3x? Gemma 4, DeepSeek and NVIDIA Compared

What is Multi-Token Prediction That Speeds Up LLMs 3x? Gemma 4, DeepSeek and NVIDIA Compared

Every developer working with Large Language Models (LLMs) shares the same complaint: the model gives great answers, but producing them takes far too long. Especially for models above 30 billion parameters, generating a single token requires moving billions of parameters from VRAM to compute units. This makes standard autoregressive generation memory-bandwidth bound, creating a serious latency bottleneck.

In this article, we examine Multi-Token Prediction (MTP), which Google announced for the Gemma 4 family. What is this technology, how does speculative decoding work, and how does it compare to leading models like DeepSeek V3, Qwen3-Next, and NVIDIA Nemotron 3 Super? Let's dive in.

What is Multi-Token Prediction (MTP)?

In standard autoregressive generation, the model predicts only the next token at each step. It looks at the context, produces one token, appends it, and repeats. This chaining is inherently serial with limited parallelization.

Multi-Token Prediction (MTP) adds training targets that ask the model to predict multiple future tokens, not just the next one. As Sebastian Raschka explains in his LLM Architecture Gallery, the model receives richer supervision through additional prediction heads for t+1, t+2, t+3 at each position.

This encourages the model's hidden states to encode information useful for future tokens, not just the immediate next one. In other words, the model learns to plan ahead.

Speculative Decoding: The Speed Secret

MTP's real power emerges during inference through speculative decoding, a method defined in Google's "Fast Inference from Transformers via Speculative Decoding" paper. It relies on collaboration between two models:

1. Lightweight draft model: A much smaller and faster model (or the model's own MTP head) works alongside the main model, using idle compute to predict several future tokens at once.
2. Main model verification: The heavy, authoritative main model (e.g., Gemma 4 31B) verifies all proposed tokens in parallel in a single forward pass.
3. Accept or reject: If the main model agrees with the draft, it accepts the entire sequence at once, and even generates an additional token. Thus, multiple tokens are produced in the time normally needed for one.

According to Google's own explanation, if the target model agrees with the draft, "it accepts the entire drafted sequence in a single forward pass, and even generates an additional token of its own. This means your application can output the full drafted sequence plus one token in the time it usually takes to generate a single one."

Google Gemma 4 MTP Drafters: 3x Speedup

In May 2026, Google announced Multi-Token Prediction drafters for the Gemma 4 family. This came after Gemma 4 surpassed 60 million downloads, and the drafters are released under the same Apache 2.0 open-source license.

The performance claims for Gemma 4 MTP drafters include:

• Up to 3x speedup: Tested across LiteRT-LM, MLX, Hugging Face Transformers, and vLLM on various hardware.
• Zero quality degradation: The main model retains final verification authority, so output quality and reasoning remain unchanged.
• 2.2x speedup on Apple Silicon: The 26B MoE model at batch sizes 4-8 (vs. batch size 1).
• Similar gains on NVIDIA A100: Batch size scaling is effective on large GPUs too.
• Half the wait time: On NVIDIA RTX PRO 6000 for Gemma 4 26B, with identical output quality.

Supported models include Gemma 4 31B (Dense), Gemma 4 26B (MoE), and edge variants E2B, E4B. Developers can access them via Hugging Face, Kaggle, transformers, MLX, vLLM, SGLang, and Ollama.

What Are Competitors Doing? The MTP Ecosystem Compared

MTP is not unique to Gemma 4. Over the past few years, many leading models have adopted multi-token prediction or speculative decoding in different forms:

DeepSeek V3: MTP-1 for Training and Inference

DeepSeek V3 adds one extra token prediction during training through its MTP-1 structure. This helps the model learn richer future-oriented representations. During inference, this MTP module can optionally be used for speculative decoding. The DeepSeek team reports positive effects on coding and math capabilities.

Qwen3-Next: High-Acceptance MTP Module

Alibaba's Qwen team optimized the MTP module in Qwen3-Next 80B-A3B specifically for speculative decoding. The focus is on increasing the acceptance rate of draft tokens by the main model.

NVIDIA Nemotron 3 Super: Internal Draft Model

NVIDIA's Nemotron 3 Super 120B-A12B uses a shared-weight MTP head that acts as an internal draft model. This enables native speculative decoding without requiring a separate external draft model, making deployment and integration highly practical.

Step 3.5 Flash: Both Training and Inference

Step 3.5 Flash 196B uses an MTP-3 configuration during both training and inference. This is less common, as MTP is mostly used as a training signal. Step's dual-phase approach is notable.

MiniMax M2.7, Xiaomi MiMo-V2, Tencent Hy3

In the Chinese ecosystem, models like MiniMax M2.7 230B, Xiaomi MiMo-V2-Flash 309B, and Tencent Hy3-preview also use MTP or similar multi-token prediction structures. This shows MTP is becoming a standard, not a luxury, for large-scale LLMs.

Model	MTP Configuration	Use Case
Gemma 4	MTP Drafters (external)	3x speedup, open source
DeepSeek V3	MTP-1 (optional inference)	Coding and math
Qwen3-Next	MTP module (high acceptance)	Edge and cloud optimization
Nemotron 3 Super	Shared-weight MTP head	Native speculative decoding
Step 3.5 Flash	MTP-3 (training + inference)	Speed and quality balance

Real-World Performance and Limitations

MTP and speculative decoding do not perform uniformly across all scenarios. According to Nexus team's evaluation, acceptance rate depends on how predictable the output is.

High acceptance rate scenarios:

• Code generation: syntax is highly structured.
• Structured data extraction: JSON, CSV, templated outputs.
• Standard text patterns: contract language, repetitive phrases.

Lower acceptance rate scenarios:

• Open-ended dialogue and creative writing.
• Heavy reasoning tasks with non-obvious chain-of-thought.
• Adversarial inputs where the model is already uncertain.

For example, Gemma 4 MTP delivers ~18% throughput improvement in structured data extraction, but only ~11% in open-ended summarization. In some cases, topic drift rate increases slightly. Therefore, benchmarking MTP for your specific use case is critical.

Technical Infrastructure: KV Cache Sharing and MoE Optimizations

The engineering behind Gemma 4 MTP drafters is also noteworthy. Draft models reuse the target model's activations, and KV cache sharing eliminates redundant context recalculation.

For E2B and E4B edge models, where final logit calculation is a bottleneck, clustering techniques further accelerate generation. For the 26B MoE model on Apple Silicon, unique routing optimizations at batch size 1 were applied.

These details show MTP is not just an algorithm but an optimization requiring hardware-model collaboration.

Practical Guide for Developers

If you want to try Gemma 4 MTP drafters, here are quick starting points:

1. Documentation: Google's official MTP documentation provides the most up-to-date integration guide.
2. Framework support: Works with transformers, MLX, vLLM, SGLang, Ollama, and LiteRT-LM. No need to manage a separate draft model in your existing pipeline.
3. Mobile testing: The Google AI Edge Gallery app (Android and iOS) lets you test MTP on edge models directly on your phone.
4. Use-case analysis: If your application is code generation or structured output heavy, you will get maximum benefit. For creative writing or open-ended chat, gains will be more modest.

Conclusion: Speed is Accessibility

Multi-Token Prediction is a technique poised to solve one of the biggest practical problems of LLMs: latency, without sacrificing quality. Google's Gemma 4 MTP drafters make this technology accessible to everyone with up to 3x speedup and an Apache 2.0 license.

But MTP is not a magic wand. Its performance varies by use case, so it should be evaluated with proper benchmarks and real-world testing. For coding, structured data processing, and agentic workflows, MTP is becoming almost essential. For creative applications, it offers more modest gains.

In conclusion, speculative decoding and MTP are now standard parts of large-scale LLMs. As developers, using these tools in the right place and right way directly impacts user experience. Have you tried Gemma 4 MTP drafters? Share your experiences in the comments or contact me.

Sources:

• Google Blog: Multi-Token Prediction in Gemma 4
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding
• Gloeckle et al., Better & Faster Large Language Models via Multi-token Prediction
• Sebastian Raschka: Multi-Token Prediction
• Google AI: Gemma MTP Documentation
• Hugging Face: Gemma 4 Collection