Faster prompt lookup drafting in llama.cpp

In one line

Four changes to the n-gram caches of llama.cpp make drafting for prompt lookup decoding up to 41.6x faster per drafted token, load the static cache up to 23.5x faster, and lower peak memory up to 2.65x.

Opening excerpt

[ homepage ] [ github ] [ twitter ] This article was originally published on 2026-09-26.

TL;DR I make drafting for prompt lookup decoding in llama.cpp up to 42x faster while using up to 2.6x less memory through a set of simple performance optimizations largely based on the work of Daniel Lemire and Martin Ankerl .

Update: Daniel Lemire sent in a PR that makes prompt lookup drafting upto 4.2x faster on top of my original optimizations. His work makes the overall speedup upto 140x. I discuss more below .

Many popular inference engines including llama.cpp and vllm, and machine learning libraries such as hugging face’s transformers library, support prompt lookup decoding (also called n-gram speculation ) for faster token generation. Prompt lookup decoding is technically a special case of speculative decoding that uses a really stupid draft model, an n-gram model. When prompt lookup decoding is used, the inference engine drafts the next $k$ tokens using the following rule.

(Excerpted from the original; full article via the source link below.)

This story hit the Hacker News front page today (58 points / 9 comments, via jadidbourbaki.github.io). Our Tech Radar aggregates daily signals on AI engineering, backend architecture and DevOps — browse the related services and further reading below, or get in touch with our team.