Trimio Field Notes

Why Your AI Agent Keeps Forgetting Things

May 25, 2026 7 min read cachingagent-loopsproxyperformance

Turn 9 of a 20-turn agent session. The model starts repeating itself — asking for context it already has, re-deriving conclusions it reached three turns ago. Your agent has effectively restarted mid-session.

This isn't a model capability failure. The model hasn't "forgotten" in the way humans forget. What happened is a cache collapse: the token prefix that was cached at the provider's inference layer was invalidated, and the model received a materially different context than it expected. The session restart is the symptom. The cache collapse is the cause.

This week, DeepSeek Reasonix landed on Hacker News with 640 points for solving this problem at the agent-framework level. Their core claim: most agent loops achieve less than 20% cache hit rate because they inadvertently break prefix immutability — by reordering messages, injecting timestamps, or modifying tool call outputs between turns. Their fix is a "cache-first loop" that keeps the prefix frozen. At DeepSeek's pricing (10× cheaper on cache hits), this translates to roughly a 50% reduction in effective per-turn cost.

We solved the same problem at the proxy layer. Here's what it looks like from the infrastructure side, and why it matters regardless of which agent framework you're running.

The 80% cache miss rate problem

Essential
Most agent loops achieve <20% cache hit rate in practice — not because prefix caching doesn't work, but because the proxy or compression layer modifies messages inside the cached prefix between turns, invalidating it silently.

Prefix caching works by storing the KV (key-value) computation for the first N tokens of a context. When the next request starts with those exact same tokens — same bytes, same order — the provider reuses the cached computation instead of recomputing from scratch. For DeepSeek V4, that's a 10× cost difference. For Anthropic, it's roughly 10× on cache hits vs. misses on input tokens.

The problem is prefix immutability. Prefix caching only works if the prefix is genuinely frozen between turns. Any modification — even a minor one — invalidates the cache and forces a full recomputation.

Common ways agent loops break prefix immutability without knowing it:

In production, the failure mode is subtle. Token cost goes up (cache misses are expensive). Session quality degrades (the model loses accumulated context). The agent loop appears to "forget" — but what actually happened is that the proxy infrastructure was silently rewriting the context that should have been frozen.

The real customer incident

Essential
Tony's production session: 88k → 23k cached read tokens (73% cache loss) on turn 9 of a Fireworks/Qwen session. Root cause: Trimio's compression layer was eligible to rewrite messages inside the cached prefix because the session anchor was only guarded for Anthropic requests.

Earlier this week, one of Trimio's proxy-test users — call him Tony — ran a 20-turn session on a Fireworks/Qwen3 backend. Fireworks uses server-side prefix caching with pricing similar to DeepSeek: cache hits cost roughly 10× less than fresh reads.

On turn 9, Tony's agent entered a loop. The model was asking "Let me check..." followed by the same lookup it had completed four turns earlier. Seven near-identical lookup messages in 90 seconds. Session quality tanked.

The request logs told the story:

Root cause: Trimio's compression layer (specifically the deduplication features — ToolDedup, SubstringDedup, ContextDedup, SemanticDedup) was rewriting messages at indices 2, 5, 6, 7, 8 inside the cached prefix. These rewrites were correct from a compression standpoint — they reduced token count. But they changed bytes inside the prefix, collapsing the cache on turn 9.

The underlying bug: the session anchor machinery (which tells the compression layer "don't touch messages inside this prefix") was gated on IsAnthropicRequest(). For Fireworks, Gemini, and OpenAI — all of which use implicit server-side prefix caching — the anchor gate never fired. Every message was eligible for compression, including messages inside a live cached prefix.

The fix: session anchor at the proxy layer

Essential
The fix: extend the session anchor to all implicitly-cached providers (Fireworks, Gemini, OpenAI) and make every compression feature check the anchor floor before modifying any message. Messages inside a live cached prefix become ineligible for content rewrites.

The fix required three PRs:

  1. PR #624: Added isImplicitCachedProvider() helper — returns true for Fireworks, Gemini, OpenAI (all providers with automatic server-side prefix caching). The session anchor machinery now calls SetImplicitAnchor() after each turn for all four provider families, not just Anthropic.
  2. PR #623: Updated all four compression dedup features to check the anchor floor before modifying any message. Messages with index ≤ anchor floor are ineligible for content rewrites. Compression still applies to messages above the anchor floor.
  3. PR #622: Groundwork for the anchor data model to support multiple provider families simultaneously.

After these PRs deployed, Tony's session type — a long multi-turn Fireworks session with heavy compression — would maintain its cached prefix for the full session duration. Cache hit rate for equivalent sessions jumped from ~40% (unreliable, anchor missing) to ~90%+ (stable, anchor protecting the prefix). Per-turn cost dropped accordingly.

Why the proxy layer is the right place to fix this

Essential
DeepSeek Reasonix solves cache stability at the agent framework level — you rebuild your agent loop. The proxy approach solves it transparently for any agent, any framework, any provider, without touching application code.

DeepSeek Reasonix is a well-designed solution to a real problem. Their cache-first loop design is elegant. But it requires you to build your agent with Reasonix's architecture, and it targets DeepSeek backends specifically.

The proxy approach is different:

The Reasonix approach and the proxy approach are complementary, not competing. Reasonix optimizes your agent loop for cache stability. The proxy handles the compression and routing layer that sits below the agent loop. You can run both.

The cost math

For a session with 70k context tokens and 10k new tokens per turn:

Without cache protection
$0.16/turn
80%+ cache miss rate, fresh reads dominate
→
With proxy-layer cache protection
$0.04/turn
90%+ cache hit rate, anchor holds

For a 20-turn agent session on Fireworks/Qwen at these context depths:

At 1,000 sessions per day, that's $2,400/day in recoverable cost that was invisible until someone noticed Tony's agent loop.

How to tell if you have this problem

Essential
Check your cached_read_tokens per turn over a long session. If they drop significantly (30%+) mid-session without a context change from your application, you have a cache collapse problem. The fix is proxy-layer session anchor protection.

If your proxy or provider dashboard exposes per-turn token breakdowns (fresh vs. cached read vs. cached write), look for this pattern:

The drop can be gradual or sudden. In Tony's case it was sudden (turn 9, full collapse). In cases where compression is less aggressive, it can be a slow erosion over many turns.

The other signal: agent quality degradation mid-session that doesn't correlate with any change your application made. If your agent starts asking for context it already received, or repeating conclusions it reached earlier, check the cache token trace before assuming it's a model capability problem.

What this means for multi-provider routing

The session anchor work matters beyond Fireworks/Qwen. With Gemini response translation now complete (PRs #620/#621), LCR routes to Google Gemini backends return responses in the client's native format (Anthropic or OpenAI). Combined with the session anchor extension, a route that fails over from Anthropic to Gemini — or from Fireworks to OpenAI — now preserves cache stability across the provider switch.

Without the session anchor: a failover event collapses the cache (the new provider doesn't have the cached prefix). With it: the anchor is provider-aware, and the compression layer doesn't touch the frozen prefix regardless of which provider receives the request.

Multi-provider routing is only useful if the session quality holds across providers. Cache stability is a prerequisite for that.

The bottom line

Essential
If your AI agent appears to "forget" mid-session, the cause is almost certainly a cache collapse — not a model capability failure. The fix is proxy-layer session anchor protection that keeps the cached prefix immutable regardless of compression or routing operations running below your application.

Agent memory failure is an infrastructure problem, not a model problem. The model behaves consistently when its context is stable. The proxy and compression layers that sit between your application and the model can silently destabilize that context — unless they're designed with cache prefix protection as a first-class constraint.

DeepSeek Reasonix approached this from the agent-framework direction and got 640 HN points for it. The problem is real and the community knows it. The proxy-layer approach solves it transparently, for any agent, any framework, any provider — without requiring you to rebuild your agent loop.

Trimio is the LLM API gateway with built-in session anchor protection, multi-provider cache stability, and per-request token cost visibility. One URL change. No agent modifications required. See how it works.

Trimio
Stop guessing. Start governing.
trimio is the LLM API gateway purpose-built for AI cost governance — visibility, routing, caching, and budget enforcement in one layer.