How Does LLM Caching Work? Reading AI Infrastructure Efficiency Through Anthropic’s S-1

Wed, 28 Oct 2026 10:00:00 +0900

Anthropic’s S-1 describes reductions in the inference compute needed to deliver a given level of model performance, alongside improvements in how efficiently that compute is used in production. – Summary of “Compute Efficiency”¹

Dohyun Jung - Principal Consultant, ROBOCO

TL;DR

LLM infrastructure is evolving to require less compute for a task at a given performance level and to make better use of available resources. Anthropic’s discussion of inference efficiency in its S-1 is one piece of evidence for that trend.
Prompt caching reuses the KV cache produced while processing an identical input prefix. It reduces repeated computation over long documents, while new questions still need processing and answers still need generating.
At scale, systems share common computations in memory blocks and route requests with both cache location and load in mind. Some also store caches outside GPU memory.
Improving cache efficiency starts with deciding whose computations may be shared with whom. User isolation can reduce reuse, and the same authorization boundaries must follow caches outside the GPU.
The benefits depend on reuse frequency, available memory, and transfer costs. The reduction in reprocessed input tokens must not be treated as the reduction in total cost.

Imagine visiting a support specialist with the same product manual. On the first day, the specialist has to read it from the beginning. You return the next day with a different question, but the specialist starts reading the entire manual again. Repeated reading could take longer than answering the question.

Ai-Infrastructure on ROBOCO

How Does LLM Caching Work? Reading AI Infrastructure Efficiency Through Anthropic’s S-1

Dohyun Jung - Principal Consultant, ROBOCO

TL;DR