Speculative Decoding for 400% Faster LLMs

DeepSeek’s new DSpark module brings speculative decoding to DeepSeek-V4. It might look like a niche inference tweak, but in production it boosted per-user generation speed by 60 to 85 percent with no drop in model quality.

What sets DSpark apart is that it tackles two longstanding problems at once, weak draft quality and the waste of verifying drafts, where prior methods addressed only one. In this article, I’ll break down how it solves both and why that matters at production scale.

What Is Speculative Decoding?

LLM generation is slow because each token needs a full forward pass through the model. Speculative decoding speeds this up with a smaller draft model that predicts several future tokens at once, which the target model then verifies in a single pass.

If the draft model makes good predictions, several tokens can be produced from a single forward pass through the target model. If it makes poor predictions, it reverts to its normal pace. Output quality is still maintained because the target model verifies the predictions against its own probability distribution.

The key issue is developing an appropriate draft model:

• When it is sequential and accurate over long predictions, it cannot keep up with the target model and fails to produce multiple tokens before the target finishes.
• In this case, latency keeps increasing based on the number of blocks being processed.

By making the draft model faster and parallel rather than sequential, the predictions become less accurate in the latter part of the block. DSpark demonstrates a solution that addresses both factors at once.

The Core Idea: Semi-Autoregressive Drafting

Here is a pattern of predictive modeling: in an autoregressive context (i.e., Eagle3), each generated token is conditioned on all previously generated tokens. While this is representative of traditional machine learning training, it is inefficient, since the model experiences a linear increase in latency the longer the number of tokens generated.

In a parallel context (i.e., DFlash), the model generates an entire block of tokens in a single forward pass. This produces very fast output. However, each token is estimated in isolation from the others positioned in the block. As you can imagine, the output from such a model can create an odd mix of words. Take “of” and “problem” as an example: each forms a reasonable phrase (“of course” and “no problem”), but used together (“of problem”) they no longer make any sense.

DSpark combines a largely parallel structure for speed (many independent processing paths) with a tiny sequencing structure that adds local dependencies between tokens. Together, it’s a mostly-parallel approach with a thin layer of autoregression on top to fix incoherence across the sequence.

The paper presents two sequencing structures:

• A Markov head uses only the preceding token plus a low-rank matrix, achieving nearly no overhead.
• An RNN head maintains a minimal recurrent state across the block, giving it more context than the Markov head.

DeepSeek found the Markov head delivers essentially all the benefits at much lower complexity, so that’s the one they put into production.

Getting Started with DeepSpec

DeepSeek has open-sourced the training and evaluation code for their draft models as DeepSpec. This is a complete repo to train any type of draft models, and not just for DSpark, but also for DFlash and Eagle3. You can reproduce their comparisons of those models using this repo. 

To install the dependencies and clone this repo, see the README files included in the repo. 

git clone https://github.com/deepseek-ai/DeepSpec.git
cd DeepSpec
python -m pip install -r requirements.txt 

This covers the installation for training and evaluating models with DeepSpec. However, you will still need to prepare your data separately by using a mechanism to infer outputs from the target model. For more information about how to do that, consult the scripts/data/README.md within this repo. 

Hands-On: Training and Evaluating a Draft Model

There are three stages in a DeepSpec workflow: preparing data, training your model from the draft, and evaluating it. The output of one stage becomes the input to the next stage. 

Step 1: Picking a Config 

You can find configs in the config/ folder (there is one file for every pair of algorithms and target models). 

ls config/dspark/
# dspark_qwen3_4b.py  dspark_qwen3_8b.py  dspark_gemma4_12b.py 

Each config file specifies the Target Model, Block Size, and which sequential head should be used. If you want your set up to be the same as the smallest benchmark described in the paper, then you will want to use the dspark_qwen3_4b.py configuration file. 

Step 2: Training the model 

To start training, you will use the following command:  

bash scripts/train/train.sh --opts config_path=config/dspark/dspark_qwen3_4b.py 

The script may create a worker for each GPU that is in your system. Checkpoint files will be saved in ~/checkpoints///step_*. If you are only using a single node for training, you will need to set the CUDA_VISIBLE_DEVICES variable to match the number of GPUs you have. 

Within the training process itself, we are optimising three loss types at the same time: 

• a cross-entropy term (for predicting the next token correctly),  
• a distribution-matching term (which directly relates to the “acceptance rate” of the generated content), 
• a “confidence loss” 

This last one is important, as it allows us to implement the scheduling trick described in the next section. 

Step 3: Evaluation 

bash scripts/eval/eval.sh \ 
  --target_name_or_path Qwen/Qwen3-4B \ 
  --draft_name_or_path 

~/checkpoints/deepspec/dspark_block8_qwen3_4b/step_latest

Verification happens in one pass, so measure how many tokens are accepted across three task types: math, code, and chat. More accepted tokens means fewer wasted forward passes toward the target model.

Experimental Results

The figures presented by DeepSeek were notable indeed. DSpark exceeded Eagle3’s accepted length by about 27-31%. DSpark’s output exceeded DFlash by 16-18%. Both improvements remained consistent across all the Qwen3-4B, 8B, and 14B targets. Furthermore, they performed similarly on the Gemma4-12B as well, indicating that there is also something with Gemma’s results and not just a quirk of Qwen’s.  

The cross-family outcome helps to clarify why DeepSeek’s post had the titles of both Gemma and Qwen listed. This should be viewed as a better indication than comparing only to a single model. Architecture-specific tricks usually break down when tested on an alternate division of models. 

Gotchas and Things That Trip People Up

Here are some pieces of information that are very important, no matter what form the information is presented in: 

• Chat Verifies Unlike Code: Chat has more valid next tokens (meaning it has a lower confidence rate) than code, so confidence decreases faster and scheduling will prune more aggressively. 
• Static Thresholds are Not Dynamic Scheduling: A static cutoff is last year’s technology, and the cutoff does not consider how busy your system is, DSpark will recalculate a dynamic cutoff each batch. 
• Causality is non-negotiable: Because you cannot see into the future, the scheduler cannot check a token before it verifies that the token has been validated. This is often managed off-line using the two-step confidence prediction process that was in-work at the end of V2. 
• At the extreme ends of the nominal percentages are very misleading: For instance, the 661% multiplier for MTP-1@V4-Flash is under artificial conditions, the metric does not reflect a manufacturer’s real-world production so do not use the multiplication as an expected value instead use the 60-85% matched throughput. 
• You cannot recover Drafting costs: Even if your query is not accepted and you still pay a full drafting fee at the time of the query, even if the system prunes verification after scheduling. 

Conclusion

DSpark is a solid reminder that inference speedups can come from many places. Not every gain requires a bigger model or better hardware; sometimes it comes from admitting that drafts may be inaccurate and letting the scheduler work around that admission intelligently.

If you’re running speculative decoding under varying request loads, the idea applies even if your architecture isn’t DeepSeek-like. The premise is simple: only verify what has positive expected value.

And if you’re wondering how the Markov head stacks up against full attention for the draft block, that’s the next rabbit hole to chase. You can test it yourself, since the DeepSpec repo has everything you need.

Frequently Asked Questions