Does Karpathy's LLM Wiki Actually Work? A 72-Run Benchmark

Thu, 07 May 2026 11:00:00 +0900

“To keep agents from re-deriving everything every time, you need a compiled knowledge artifact.” — Andrej Karpathy, LLM Wiki (GitHub Gist, 2026-04)¹

Dohyun Jung - Principal Consultant, ROBOCO

TL;DR

We measured whether Andrej Karpathy’s LLM Wiki pattern (“don’t re-read every time; reuse knowledge compiled once”¹) is actually a win, using an 8-task × 3-technique × 3-repetition = 72-run benchmark in a 30-repository migration workspace.
LLM Wiki placed first on all three dimensions: tokens, time, and quality. Versus Vanilla, 54% fewer tokens and 39% less time (statistically a large effect, §4), with quality equal to Vanilla and better than Graphify.
Graphify (GraphRAG) is unsuitable as a default tool. Negligible token savings, the longest time, and the lowest quality.
That said, it’s not universal. Wiki wins decisively on tasks that synthesize multiple sources, but for tasks whose answer sits plainly in a single source (exhaustive grep, looking up a specific table), reading directly is faster and more accurate (§3).
On receiving these results, the project’s default context-retrieval tool was switched to LLM Wiki that same evening.

Technique	Tokens (avg)	Time (avg)	Quality (out of 25)
Vanilla (direct read/grep)	755K	167s	15.1
LLM Wiki	350K	101s	16.0
Graphify (GraphRAG)	617K	180s	13.6

Below is the experimental design and detailed numbers that support this conclusion.

Benchmark on ROBOCO

Does Karpathy's LLM Wiki Actually Work? A 72-Run Benchmark

Dohyun Jung - Principal Consultant, ROBOCO

TL;DR