Does Karpathy’s LLM Wiki Actually Work? A 72-Run Benchmark

“To keep agents from re-deriving everything every time, you need a compiled knowledge artifact.” — Andrej Karpathy, LLM Wiki (GitHub Gist, 2026-04)¹

TL;DR

• We measured whether Andrej Karpathy’s LLM Wiki pattern (“don’t re-read every time; reuse knowledge compiled once”¹) is actually a win, using an 8-task × 3-technique × 3-repetition = 72-run benchmark in a 30-repository migration workspace.
• LLM Wiki placed first on all three dimensions: tokens, time, and quality. Versus Vanilla, 54% fewer tokens and 39% less time (statistically a large effect, §4), with quality equal to Vanilla and better than Graphify.
• Graphify (GraphRAG) is unsuitable as a default tool. Negligible token savings, the longest time, and the lowest quality.
• That said, it’s not universal. Wiki wins decisively on tasks that synthesize multiple sources, but for tasks whose answer sits plainly in a single source (exhaustive grep, looking up a specific table), reading directly is faster and more accurate (§3).
• On receiving these results, the project’s default context-retrieval tool was switched to LLM Wiki that same evening.

Below is the experimental design and detailed numbers that support this conclusion.

1. The Three Techniques Compared

The core of the LLM Wiki pattern is its difference from RAG. RAG is a stateless cycle that repeats embedding, vector search, and chunk injection on every query, so it redoes synthesis, cross-referencing, and contradiction handling from scratch each time. LLM Wiki inverts this: it compiles once when a new source arrives (summarizing, cross-referencing, flagging conflicts), and at query time it reads a markdown page that is already synthesized. Compile once, query many times — this asymmetric cost allocation is the source of the token savings.

2. Experimental Design

Decision question: “For this migration workload, which technique satisfies both token efficiency and answer quality?” The dependent-variable priority was (1) token cost, (2) answer quality, (3) task time.

Workload — 8 tasks (T1–T8):

Isolation: the three techniques were separated into 3 git worktrees, each with a dedicated CLAUDE.md overwritten in place. Submodules, PRD, code, and git were shared; the only difference was the auxiliary tooling. Thanks to the separate worktree paths, there was no prompt cache contamination across trials.

Scoring (Judge) — an external model (codex/GPT-5): since the actor was Claude, we scored with an external model to avoid self-evaluation bias. A 4-dimension 0–25 rubric.

The task prompts did not expose ground truth; only the judge held it. Inter-rater agreement (Cohen’s κ) passed the rubric at a weighted average of 0.86 (only the excess dimension was weak at κ=0.25, but its weight is 0.1).

3. Winners by Task — The Pattern Isn’t Simple

The overall average favors Wiki, but the winner varies by task type. First place by mean score:

• Wiki is robust on mixed and synthesis-type tasks. First place in 5 of 8 (T1, T2, T5, T6, T7). It’s especially strong on T2 (cross-language) and T6 (external sources), where multiple sources must be synthesized. Curation earns its keep.
• There are two tasks where Vanilla wins: T4 (exhaustive grep) and T8 (cross-domain). In both, the answer sits plainly in a single source, so curation is a net loss. T4 is settled by grepping the whole codebase, and for T8 reading the PRD table directly is the most accurate route.
• Graphify took first place on only one task, T3, and even there at 9.50/25 the absolute score is low. It also lost badly to wiki on cross-language (T2), where we expected it to shine — INFERRED edges added noise and query overhead inflated the time.

4. Statistical Significance

Friedman main effects: time p=4.5e-07, quality p=0.0016 (both strong), tokens p=0.093 (marginal).

Wilcoxon pairwise (Bonferroni-corrected):

In summary, Wiki cuts tokens and time versus vanilla with a large effect (quality equal), and beats graphify on both quality and time. Graphify-first is unsuitable as a default tool for our workload.

5. So How Do We Operate?

This is the baseline policy we settled on after receiving the benchmark results.

1. Wiki-first — for operational facts, read wiki/index.md → the relevant notes first.
2. If it’s not there, go straight to the source — if the wiki doesn’t have it, PRD → code read/grep. Never make anything up.
3. Skip the wiki when single-source retrieval is obvious — if it doesn’t narrow to a single note, move quickly to a direct read (the T4/T8 shape).
4. Graphify is auxiliary — use it selectively, only for cross-component dependency tracing.

The biggest lesson from operating this is a separate one. Getting answers to cite their sources is harder, and more important, than building the wiki. Early on, note-taking was active but fewer than 10% of answers carried [[..]] citations, so only half the wiki’s value materialized. A single check rule forcing “is there a wiki citation for this fact?” right before answering brought the citation rate back to normal immediately.

6. Caveats

• IRR on the excess dimension is weak (κ=0.25). Because scoring was done while viewing all techniques’ transcripts together, excess scores were uniformly harsh. With a weight of 0.1 the impact on the conclusion is small, but treat the absolute values conservatively.
• Concern about Wiki bias: all 8 tasks have ground truth present in the wiki, which may favor it. That said, the judge evaluates the ground truth facts themselves rather than the source.
• This is a Claude Code-based measurement. Results may differ in other model and tooling environments. These numbers are the basis for policy in our environment, not a universal claim.

Wrap-Up

In our domain, Karpathy’s LLM Wiki pattern delivered 54% fewer tokens, 39% less time, and quality equal or better, and was statistically significant with a large effect. The decision to switch our default tool to the wiki is backed by data.

That said, don’t apply the pattern literally. On tasks whose answer sits plainly in a single source, the cost of curation is a loss, and while Graphify is unsuitable as a default tool, it beats the wiki on certain path queries. Adopt the pattern, but look at it alongside your own workload’s task distribution and source structure. At 8 tasks × 3 techniques × 3 trials = 72 runs, you’ll have an answer within a day or two, so I recommend measuring it yourself.