GOOGLE TRIPLES GEMMA 4 SPEED WITH MULTI-TOKEN PREDICTION

Google's optimization technique addresses a fundamental bottleneck in large language model inference. Traditional token-by-token generation requires the main model to process output sequentially, creating latency constraints even for efficient architectures. The multi-token prediction approach splits the workload between two components. A lightweight auxiliary model predicts several upcoming tokens in parallel, functioning as a draft generator. The primary Gemma 4 model then evaluates all proposed tokens in one computational pass, either accepting or rejecting predictions before proceeding to the next set. This method achieves up to 3x speedup across the Gemma 4 open model family while maintaining output quality. The technique proves particularly effective for inference-constrained scenarios where latency directly impacts user experience. The drafting strategy mirrors speculative decoding approaches explored by other labs, but Google's implementation targets open-source accessibility. Gemma 4 models span multiple sizes, making the speedup relevant for deployment across consumer hardware to data center infrastructure. No additional training or fine-tuning of existing Gemma 4 checkpoints is required. The auxiliary drafting models are released alongside the main model weights, enabling immediate integration into existing inference pipelines. The optimization carries implications for real-time applications including chatbots, code generation, and streaming text interfaces. Reduced latency lowers computational costs per request while improving perceived responsiveness. Google has not disclosed whether this technique will extend to Gemini or other proprietary models. The release focuses on expanding Gemma's competitive positioning within the open-source LLM ecosystem, where performance-per-compute has become a primary differentiation metric. The multi-token prediction drafters are available through Google's official Gemma releases, with integration documentation for frameworks including JAX and PyTorch.