DLite: History-Free Speculative Decoding for Agents

Background and Context

Speculative decoding speeds up large language model inference in a simple way. A light draft model proposes a run of candidate tokens. The target model then verifies them in one parallel pass. Each accepted token saves one sequential generation step. The key metric is acceptance length, the number of draft tokens the target model accepts per verification round.

Why does speculative decoding fail on agent workloads? According to the DLite paper on arXiv, agent contexts are long, they keep evolving, and they span many interaction turns. Under these conditions the acceptance length of existing draft models drops by up to 80%. The paper also finds that keeping the history KV cache for the draft model adds heavy system cost. So the technique loses speed and gains overhead at the same time.

The authors trace both problems to one cause: the growing history context. This diagnosis shapes the whole design. They do not build a bigger or smarter draft model. They ask whether the draft model needs the history at all. The answer is DLite, a history-free speculative decoding framework built for agents.

Deep Analysis

DLite has three working parts. The first is reuse of the target model's contextual hidden states. The target model must process the full context during verification anyway. Its hidden states already carry a compressed view of that history. The draft side can read them directly. It does not need to build or store a separate history representation, and it keeps no draft-side history KV cache. The second part is local speculative prediction. The draft side no longer models a context that grows with every turn. It predicts only from the current local position. In the paper's framing, a growing modeling problem becomes a simple local one. This also explains the system gain: with no explicit history to maintain, the cache cost the paper describes does not arise on the draft side. The third part is a lightweight sequential head. Local prediction has a weakness. The candidate tokens inside one block can lose their causal links to each other. The sequential head restores these dependencies at low cost, so a later token in the block can take account of the tokens guessed just before it. The design keeps block-level drafting but repairs its main flaw. The pieces come from different sources, so training needs care. Target-model hidden states feed a draft-side local predictor, which makes the structure heterogeneous. The authors therefore propose a two-stage training strategy fitted to the history-free design. The source material does not give the details of each stage. It also gives no ablation figures, so the separate gain from the sequential head or from the two stages cannot be stated here.

The evaluation covers three workload types: general, long-context and agent. This mix tests two things. It checks that the method does not regress on ordinary tasks. It also checks that acceptance length holds up where history is long. DLite reaches acceptance lengths of 2 to 6 across these workloads. It improves on DSpark, the state-of-the-art baseline named in the paper, by up to 70%. The summary does not report model sizes, per-dataset numbers or end-to-end latency, so those details need the full paper.

Industry Impact

The main lesson is about where the bottleneck sits. Earlier work often tried to make draft models larger and more accurate, which raises their own cost. DLite goes the other way. It makes the draft side simpler and more local, and it borrows computation the target model has already done. For agent serving, where one session may run many tool calls over a long, changing context, this trade looks well matched to the problem.

For inference engineers, dropping the draft-side history KV cache matters in practice. A smaller cache footprint and less cache management should make long sessions easier to serve. It may also make integration with existing inference engines simpler. These are expected benefits from the design, and the paper's summary does not quantify them in memory or latency terms.

For researchers, the 80% drop in acceptance length is a warning about benchmarks. A speculative decoding method that looks strong on short chats can behave very differently on agent traces. The DLite evaluation spans general, long-context and agent workloads, and that setup is a better test of whether a method will survive in real agent products.

Outlook

Several questions remain open. The first is generality. DLite reuses hidden states from the target model, so it ties the draft side closely to the target. It is not yet clear from the source how easily the approach moves across target models with different architectures. The second is the sequential head. It already restores block-level causality at low cost, and the next question is whether its overhead can shrink further without hurting acceptance length.

The third question is end-to-end value. Acceptance length of 2 to 6 is a strong signal, but users feel latency and throughput. Real multi-tool agent systems add tool wait time, variable turn lengths and batching effects. The most useful follow-up evidence would be wall-clock speedups measured in such systems, set against DSpark and other baselines on the same hardware.

For now, DLite offers a clear design principle for agent-era inference: do not make the draft model carry the history. Let the target model supply the context, and let the draft model predict locally. If the full paper confirms the gains across model families, history-free drafting could become a standard option for serving long-running agents.