Faster prompt lookup drafting in llama.cpp

一句话概要

Four changes to the n-gram caches of llama.cpp make drafting for prompt lookup decoding up to 41.6x faster per drafted token, load the static cache up to 23.5x faster, and lower peak memory up to 2.65x.

原文开头节选

[ homepage ] [ github ] [ twitter ] This article was originally published on 2026-09-26.

TL;DR I make drafting for prompt lookup decoding in llama.cpp up to 42x faster while using up to 2.6x less memory through a set of simple performance optimizations largely based on the work of Daniel Lemire and Martin Ankerl .

Update: Daniel Lemire sent in a PR that makes prompt lookup drafting upto 4.2x faster on top of my original optimizations. His work makes the overall speedup upto 140x. I discuss more below .

Many popular inference engines including llama.cpp and vllm, and machine learning libraries such as hugging face’s transformers library, support prompt lookup decoding (also called n-gram speculation ) for faster token generation. Prompt lookup decoding is technically a special case of speculative decoding that uses a really stupid draft model, an n-gram model. When prompt lookup decoding is used, the inference engine drafts the next $k$ tokens using the following rule.

(以上为原文节选,完整内容见下方”原文来源”)

这条动态今日登上 Hacker News 首页(58 赞 / 9 评论,来源 jadidbourbaki.github.io)。技术雷达每日自动聚合 AI 工程、后端架构、DevOps 方向的前沿动态;相关工程落地可浏览下方的相关服务与延伸阅读,或直接与我们团队交流。