Summary
Speculative decoding uses a smaller draft model to propose several tokens, then asks the target model to verify them together. Correct verification can preserve the target model’s output distribution, but speedups—including a potential 2×—are workload-dependent, not guaranteed.
Quick Verdict
Try speculative decoding if you run a local autoregressive model, have spare memory for a draft model, and your workload produces enough accepted draft tokens to offset verification overhead. It is most promising for single-user or low-concurrency generation where decoding speed is the bottleneck. Keep ordinary decoding as the baseline for high-concurrency serving, workloads with poor draft agreement, or systems already constrained by memory.
The phrase “without quality loss” needs a precise qualification. With an appropriate verification and sampling algorithm, speculative decoding can preserve the target model’s output distribution. It does not make a smaller model as capable as the target, and it does not promise identical sampled text for every random seed or implementation. Under greedy decoding, correctly verified output can match the target model’s greedy choices. In either case, the target model—not the draft—remains the authority on accepted output.
How speculative decoding works
Ordinary autoregressive generation typically runs the target model once per new token. Speculative decoding changes the sequence: a cheaper draft model proposes a short continuation, and the target model evaluates that proposed continuation in a verification pass. The verifier accepts the proposals it can justify and corrects the sequence when a proposal fails verification. Accepted tokens can be emitted together, reducing the number of sequential target-model steps.
For sampling, preserving the target distribution requires a suitable speculative sampling procedure, including correction or rejection of proposals as needed. Merely choosing a smaller model and trusting its output is not speculative decoding with a target-quality guarantee. Exact behavior also depends on the serving engine, model compatibility, sampling settings, and numerical implementation.
The performance trade-off is straightforward: a draft model consumes compute and memory, while verification consumes target-model work. The method helps when the verifier accepts enough proposed tokens to make that extra work worthwhile. A draft model that disagrees frequently can erase the benefit or make generation slower.
Feature-by-feature comparison
| Approach | How it generates | Quality relationship | Likely fit | Main cost |
|---|---|---|---|---|
| Standard target-model decoding | Runs the target model sequentially for each generated token. | Baseline target-model behavior. | General-purpose local inference; reliable baseline for comparison. | Sequential target-model steps can limit tokens per second. |
| Draft-model speculative decoding | A smaller model proposes several tokens; the target verifies them. | Can preserve the target distribution when the verifier and sampling algorithm are implemented correctly. | Good draft/target pairs and workloads with high draft acceptance. | Draft-model compute, memory, and verification overhead. |
| Prompt or n-gram lookup speculation | Reuses likely continuations found in the prompt or available context instead of relying only on a separate learned drafter. | Target verification is still required; output behavior depends on the verifier. | Repeated text, templated responses, or contexts containing useful continuations, when supported by the serving engine. | May find few useful proposals on novel or varied text. |
Practical rule Measure the target model’s throughput and latency both with and without speculation. A headline acceptance rate alone is not a performance result: end-to-end tokens per second and request latency decide whether the method helps.
Real-world workflow tests
The following are reproducible test plans, not claimed benchmark results. They specify a concrete testbed so readers can measure their own deployment without mistaking an illustrative setup for published performance data.
Testbed and controls
One practical single-GPU testbed is a desktop with an NVIDIA GeForce RTX 4090 (24 GB VRAM), an AMD Ryzen 9 7950X, 64 GB system RAM, Ubuntu 24.04, and a current CUDA-compatible vLLM installation. Record the exact GPU driver, CUDA and vLLM versions, model revisions, quantization, and power limit before testing. A different GPU, driver, model format, or serving-engine version can produce different results. The hardware listed here is a proposed reproducibility setup, not a claim that ThingDaily measured these configurations.
For each comparison, pin the target and draft revisions; use the same prompt set, target sampling parameters, context limit, batch or concurrency, and output-token limit; and warm up each configuration before recording results. Run at least 30 requests per condition after five warm-up requests. Record time to first token, end-to-end latency, output tokens per second, peak VRAM, and the engine’s reported accepted tokens per proposal. Report medians and p95 latency. Repeat the test with speculation disabled. Do not compare runs that silently change quantization, batch size, prompt length, or sampling settings.
1. Short-turn chat
Use a fixed set of 30 short conversational prompts, each with a similar context length and a 128-token generation limit. Compare single-request latency and tokens per second with the same target model under ordinary decoding and speculative decoding. This tests the interactive case, where time to first token and completion time matter more than maximum aggregate throughput. If the draft model makes a noticeable difference to VRAM use or the chat feels slower, speculation may not suit this setup even if some proposals are accepted.
2. Code completion with repeated patterns
Use a fixed set of code-completion prefixes that include both repetitive boilerplate and less predictable code. Keep the target’s stop conditions and sampling configuration identical. Measure the two prompt groups separately: repeated patterns may favor proposal methods that can exploit matching context, while novel code may not. Inspect outputs for correctness and consistency with the target’s configured decoding behavior; do not infer a quality improvement merely from faster output.
3. Long-form generation and concurrent requests
Run prompts that request up to 1,024 generated tokens, first at concurrency one and then at concurrency four, if the GPU has sufficient memory. Record per-request latency as well as aggregate output tokens per second. Longer outputs create more opportunities for accepted proposals, while increased concurrency changes the compute and memory balance. A speedup at concurrency one does not establish a speedup at concurrency four; choose settings based on the workload you actually serve.
Set it up and tune it carefully
- Establish a baseline. Serve the target model alone and save its revision, tokenizer, quantization, sampling settings, and benchmark results.
- Choose a compatible draft. Start with a smaller model from a compatible model family, and check tokenizer and architecture requirements in the serving engine’s current documentation. A smaller parameter count does not guarantee a good drafter.
- Enable speculation using the installed engine’s documented options. For vLLM versions that support this configuration form, an illustrative command is
vllm serve Qwen/Qwen2.5-7B-Instruct --speculative-config '{'method':'draft_model','model':'Qwen/Qwen2.5-0.5B-Instruct','num_speculative_tokens':5}'. CLI options and accepted configuration fields change across versions; check the official vLLM speculative-decoding documentation before using the command. Model availability, compatibility, and hardware fit must also be verified in your environment. - Tune proposal length empirically. Test a small number of proposal lengths rather than assuming that longer is better. Longer proposals can increase verification work and rejection cost when the draft is a poor match.
- Check memory headroom. Both models, runtime buffers, and the target’s KV cache use memory. Context length and concurrency can change the KV-cache requirement substantially. Watch for offloading, out-of-memory errors, or reduced batching.
- Validate target behavior. Compare against the same target model and decoding settings. For sampling, use a method documented to preserve the target distribution. For high-stakes use, add task-specific output checks; a correct decoding algorithm is not a substitute for application evaluation.
vLLM documents multiple speculative-decoding approaches, and availability can vary with its version, model architecture, and deployment configuration. Treat the command above as an example of the configuration style—not a promise that every model pair will run unchanged.
Pricing and hidden costs
Pricing verified as of Q4 2026. vLLM is open-source software released under the Apache License 2.0, so its core software fee is $0. Model-weight access and licensing depend on the selected model; check each model card and license before commercial use. Hardware, electricity, storage, and operations are separate costs. Cloud API charges are not part of local vLLM pricing and depend on the provider, model, and token volume.
| Scenario | Core software and model access | Hardware and operating cost | AI/API costs |
|---|---|---|---|
| Solo Free | vLLM: $0. Downloadable weights may be available at no charge, subject to the model’s license and terms. | Use an existing compatible computer; electricity, storage, and hardware wear still have costs. | No API token charge for local inference. Optional external API calls are billed separately by the provider. |
| Solo Power User | vLLM: $0. Check model license and any paid tooling you choose to add. | Budget for a suitable GPU and adequate VRAM, plus electricity and possible cooling or storage upgrades. Hardware purchase prices vary by region and configuration; obtain a current quote rather than relying on a universal figure. | No mandatory AI add-on or local token fee. API fallback or hosted evaluation adds provider-specific charges. |
| Team | vLLM: $0 license fee for the core engine. Review Apache 2.0 obligations and every model’s separate terms. | Budget for one or more GPUs, monitoring, deployment, maintenance, and staff time. High availability and multi-user capacity may require more hardware than a desktop testbed. | Local requests have no per-token provider bill, but external APIs, logging, and managed services may add variable costs. |
For an electricity estimate, use your measured wall draw: cost = average watts ÷ 1,000 × hours × your electricity rate per kWh. For example, a system drawing an assumed 450 W at the wall for one hour, with electricity priced at $0.15/kWh, costs about $0.068 for that hour. This is an illustration, not a measured draw or a regional tariff. Speculative decoding can reduce the time spent generating a fixed output in some cases, but its draft-model work also uses power; compare measured energy per completed task before claiming savings.
Pros, cons, and who should choose what
| Potential benefit | Trade-off or limitation | Good fit when… |
|---|---|---|
| Can reduce sequential target-model decoding steps. | Speedup depends on acceptance and verification overhead; 2× is an outcome to test, not a guarantee. | The target is decode-bound and the draft predicts its continuations well. |
| Can preserve the target model’s output distribution with a correct algorithm. | Implementation and configuration matter; a draft-only substitution is not equivalent. | You want faster inference without replacing the target model’s behavior. |
| Can run locally without per-token API fees. | Local hardware has upfront, power, memory, and maintenance costs. | You already own suitable hardware or need local processing for operational reasons. |
| Several speculation strategies may be available in supported engines. | Feature support and model compatibility vary by engine release. | You can pin versions and test your actual model and workload. |
Choose speculative decoding when your benchmark shows a useful latency or throughput gain for your own prompts and concurrency, with acceptable VRAM use. Choose standard decoding when the draft is often rejected, memory is tight, request batching is central to throughput, or a simpler deployment is more valuable than an unproven speed increase.
FAQ
Does speculative decoding really double token generation speed?
It can approach 2× in a favorable setup, but that is not a general guarantee. The result depends on how many draft tokens the target accepts, the relative cost of the two models, proposal length, hardware, context, and concurrency. Benchmark end-to-end latency and throughput against ordinary decoding on your own workload.
Does it change the target model’s answer quality?
A correctly implemented speculative sampling method can preserve the target model’s output distribution; with greedy decoding, verified output can match the target’s greedy choices. This does not improve the target model’s capabilities, and exact sampled text need not be identical across runs. Confirm the serving engine’s documented behavior and keep all target decoding settings fixed when comparing.
Next Steps & Related Reading
- Read the official vLLM speculative decoding documentation and confirm support for your installed version and model pair.
- Browse ThingDaily coverage of local LLM hardware before upgrading a workstation.
- Browse ThingDaily coverage of LLM quantization to understand another trade-off that can affect local inference performance and memory use.
Comments (0)