GLOSSARY
Speculative Decoding
A small draft model guesses several tokens; the large model checks them in one pass — faster generation without changing the answers.
Autoregressive decoding is slow because each token waits on a full forward pass of the big model. Speculative decoding lets a cheaper drafter propose a short continuation; the large model then verifies the whole proposal in parallel and keeps the prefix that matches what it would have said. Wrong guesses are discarded; accepted guesses are free speed.
It does not change the sampling distribution when implemented correctly, which is why labs ship it as a runtime trick rather than a quality trade. The win depends on how often the drafter agrees with the teacher. On repetitive or constrained output the speedup is large; on surprising code it shrinks. vLLM and many hosted APIs now hide this behind the same endpoint.