Categories: AI/ML Research

Speculative Decoding Explained: How to Get 3× Faster Inference from Any LLM

In this article, you will learn how speculative decoding works, why it produces real speed gains at the hardware level, and what modern implementations look…

The Machine Learning Practitioner’s Guide to Speculative Decoding

Large language models generate text one token at a time.

February 12, 2026

In "AI/ML Research"

Speculative Streaming: Fast LLM Inference Without Auxiliary Models

This paper was accepted at the Efficient Natural Language and Speech Processing (ENLSP) workshop at NeurIPS 2024. Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves…

October 30, 2024

In "FAANG"

Accelerating LLM Inference on NVIDIA GPUs with ReDrafter

Accelerating LLM inference is an important ML research problem, as auto-regressive token generation is computationally expensive and relatively slow, and improving inference efficiency can reduce latency for users. In addition to ongoing efforts to accelerate inference on Apple silicon, we have recently made significant progress in accelerating LLM inference for…

December 19, 2024

In "FAANG"

AI Generated Robotic Content