Token budgets, prefix reuse, and cache correctness
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Budget context and generation separately across an agent run.
- Distinguish provider prompt caching from application result caching.
- Construct cache keys that preserve tenant, version, and permission scope.
A run consumes a sequence of contexts
An agent loop repeatedly sends instructions, conversation state, retrieved evidence, and tool results to a model. Appending everything forever can make later calls far more expensive than earlier ones. If each of ten turns resends a growing history, total input is the sum of all ten context lengths, not just the final length. A token budget should therefore cover the whole run as well as each request.
In a toy run, three requests contain 2,000, 3,000, and 4,000 input tokens plus 500 output tokens each. Total input is 9,000 and total output is 1,500. If only the final request is logged, two-thirds of the input accounting disappears. Provider tokenization and usage categories differ, so measure actual usage when available and label pre-call estimates as estimates.
Reserve capacity for completing the task
A budget allocator can divide the remaining allowance among context, generation, tools, retries, and final verification. Reserving everything for planning risks reaching the final step with no capacity to explain or validate the result. Preserve critical constraints and evidence when compacting context; a shorter summary that drops a required approval condition can make the agent less reliable.
Consider a 12,000-token remaining allowance. Reserving 2,000 for a final answer and 1,000 for verification leaves 9,000 for intermediate work. That is a policy choice, not a model context-window limit. Context capacity constrains what fits in one request, whereas the run budget constrains cumulative spending. The allocator should stop, reduce scope, or return partial progress explicitly when neither a useful next action nor completion fits. Silently exceeding the budget defeats the control.
Prompt caching and answer caching do different work
Provider prompt caching can reuse computation for a matching prompt prefix. It does not mean that every future answer is copied from an earlier response. Place stable instructions and reusable context where the provider's documented caching rules can recognize them, and keep changing request-specific content outside that stable prefix where the interface permits. Confirm cache usage from reported accounting instead of assuming that repeated text produced a hit.
Anthropic's current documentation describes prefix-based caching with provider-specific breakpoints and lifetimes. Those details can evolve and should be checked during integration. Application result caching instead stores a completed retrieval result, tool result, or answer. It can skip an entire operation, but correctness depends on whether the cached result is still valid for the current actor, source version, question, and execution configuration.
A cache key is part of the data boundary
A result cache keyed only by the user's question can leak information across tenants or return an answer based on obsolete evidence. Include the dimensions that determine the result: tenant or access scope, source revision, normalized query, prompt version, model configuration when relevant, and tool behavior version. A time-to-live is useful for limiting age, but it is not a substitute for version or authorization checks.
In an original example, tenant blue asks for revenue from report version 8. A cache hit for tenant red's same wording is invalid, even if the two reports have identical filenames. After report version 9 is published, version 8's answer may be stale immediately despite a ten-minute TTL. Cache the immutable version deliberately, or invalidate when the source changes. Never equate semantic similarity with permission to reuse another user's private result.
Account for cache effects without double counting
Usage records may report total input and a cached subset. If input is 10,000 tokens and 8,000 are cached, ordinary input is 2,000, not 10,000 plus an additional 8,000. Multiply each disjoint category by its applicable dated rate. Record cache writes and reads separately when the provider bills them differently. Tool charges, storage, and model output remain distinct categories.
The example uses an application cache with a synthetic clock and exact versioned keys. It is intentionally simpler than a shared production cache: there is no eviction policy, concurrency coordination, encrypted storage, or distributed invalidation. Its tests expose two critical correctness boundaries, expiry and tenant separation. Extend those before optimizing hit rate. A high hit rate is valuable only when the reused result is valid for the current request and permitted scope.
Work through the code
The key partitions synthetic results by tenant, immutable source version, and query. Expiry is strict at the stored boundary. This is an application result cache, not an implementation of provider prompt caching. Add prompt or permission versions when those affect result validity.
import json
def key(tenant, version, query):
return json.dumps([tenant, version, query], separators=(",", ":"))
class ResultCache:
def __init__(self):
self.entries = {}
def put(self, cache_key, value, now, ttl):
self.entries[cache_key] = (now + ttl, value)
def get(self, cache_key, now):
item = self.entries.get(cache_key)
if item is None or item[0] <= now:
return None
return item[1]
cache = ResultCache()
blue = key("blue", 8, "revenue")
cache.put(blue, {"value": 120}, now=0, ttl=10)
print(cache.get(blue, 5))
print(cache.get(key("red", 8, "revenue"), 5))
print(cache.get(key("blue", 9, "revenue"), 5))
print(cache.get(blue, 10))
{'value': 120}
None
None
None
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A provider reports 12,000 total input tokens, of which 9,000 are cached, and 1,000 output tokens. Hypothetical per-million rates are 2 for ordinary input, 0.2 for cached input, and 8 for output. Calculate the charge.
Check your understanding
Two tenants ask the same question about private reports with the same filename. Which cache behavior is correct?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.