Trimio Field Notes

Your AI Compression Savings Dashboard Is Probably Lying by 5×

May 24, 2026 6 min read compressionfinopsdata-accuracy

Most AI gateway compression dashboards show you a "tokens saved" number. Most of them are computing it wrong — and not by a rounding error. We found a 5× overstatement in Trimio's own analytics aggregates. We're publishing the bug, the math, and the corrected formula, because if we got this wrong, others have too.

how compression savings is typically computed

Essential
The naive formula — original_tokens_in - prompt_tokens — breaks the moment prompt_tokens semantics change. If prompt_tokens = fresh-only (post-caching), you're attributing cached reads as compression savings. That's not savings. That's billing math you didn't write.

The naive formula for compression tokens saved is:

sum(original_tokens_in - prompt_tokens) AS compression_tokens_saved

Where original_tokens_in is the token count before compression, and prompt_tokens is what was actually sent to the provider. On the surface, this makes sense: you sent fewer tokens than you had, the delta is the savings.

The problem: prompt_tokens is an ambiguous field. Depending on the provider and schema version, it can mean:

These two interpretations produce wildly different numbers. And they're both called prompt_tokens in different implementations.

the specific bug: schema normalization changed the semantics

In Trimio's analytics pipeline, PR #615 introduced a schema normalization: prompt_tokens was standardized to mean fresh-only input tokens. This was the correct semantic — it makes the caching and compression savings columns cleanly separable. But it had a side effect on the CAGG (continuous aggregate) formula that hadn't been updated.

Before PR #615, the formula worked (approximately):

-- Old semantics: prompt_tokens = all input tokens sent on the wire
original_tokens_in - prompt_tokens = compression savings only

After PR #615:

-- New semantics: prompt_tokens = fresh-only tokens
original_tokens_in - prompt_tokens = compression savings + cached_read tokens

The formula was now attributing cached read tokens as compression savings. Those aren't the same thing at all — one is a reduction in what you sent to the provider, the other is tokens the provider served from its own cache that you never sent in the first place.

the 5× math on a concrete example

Essential
A typical cached agentic turn: 70k cache_read + 10k fresh + 20k real compression savings. Old formula reported 80k saved. Correct answer is 20k. That's a 4× overstatement — and we've seen turns where the ratio hit 5× or higher.

Take a typical heavy agentic turn with an active cache session:

original_tokens_in
100k
Before compression
→
Breakdown on the wire
70k cache + 10k fresh
Plus 20k real compression savings

With the buggy formula:

compression_tokens_saved = original_tokens_in - prompt_tokens
                         = 100k - 10k  (prompt_tokens = fresh-only)
                         = 90k  ← WRONG (includes 70k cached reads)

With the correct formula:

compression_tokens_saved = original_tokens_in - compressed_tokens_in
                         = 100k - 80k  (compressed_tokens_in = what was actually sent)
                         = 20k  ← CORRECT

The overstatement ratio in this example is 4.5×. On sessions with higher cache utilization (90k+ cached reads against smaller compression wins), the ratio climbs above 5×. The specific 5× figure comes from our median ratio across the 1,754 affected rows in the historical audit.

what negative compression savings means

The audit surfaced something else: 1,754 Anthropic rows were showing negative compression savings. That's not a rounding error. That's the formula producing a result that says "this turn cost more tokens after compression than before."

How? Anthropic's billing reports prompt_tokens inclusive of cache_write tokens in some response shapes. Cache writes are more expensive than normal input tokens — Anthropic charges ~25% of input cost as an upfront write cost, amortized across cache hits. When the old formula subtracted those write-inclusive counts from original_tokens_in, it could produce a negative delta on turns where cache was being established (write-heavy) but compression savings were modest.

These weren't real negative savings. They were a measurement artifact. All 1,754 rows were corrected to positive values after the formula fix.

the correct formula and why the column choice matters

Essential
Use compressed_tokens_in (wire-as-sent, independent of caching) not prompt_tokens (which folds in cache semantics). The correct formula: sum(original_tokens_in - COALESCE(compressed_tokens_in, prompt_tokens)). If your gateway doesn't track these as separate columns, it can't produce an accurate compression savings number.

The fix introduced in PR #616:

sum(original_tokens_in - COALESCE(compressed_tokens_in, prompt_tokens))
    AS compression_tokens_saved

compressed_tokens_in is the token count after compression, before any caching is applied — it's the literal count of tokens in the request body sent to the provider. It doesn't fold in caching semantics, which means:

The COALESCE handles rows that predate the compressed_tokens_in column — falling back to prompt_tokens for pre-compression-era records where the distinction didn't exist.

the numbers from the audit

Historical audit results across the compressed request table:

The 250 overstatement rows are what drove the headline number. These are the turns that matter for FinOps — they're the heavy sessions, the ones your CFO sees in the "top savings contributors" table. Getting them wrong by 5× means your compression ROI story is built on inflated numbers.

what to audit in your own gateway

If your gateway tracks compression savings, ask your engineering team three questions:

  1. What exactly is prompt_tokens in your schema? Is it fresh-only, or all-input? Did that definition change at any schema migration point?
  2. Do you have a compressed_tokens_in column (or equivalent) that captures what was actually sent to the provider, independent of caching?
  3. Have you ever seen negative compression savings? If so, that's the diagnostic signal — your formula is conflating caching and compression.

If you can't answer question 1 precisely, you probably have the bug. It's a common one because schema migrations happen incrementally and the analytics CAGG formulas are rarely updated in lockstep.

Trimio
Verified savings numbers, methodology published.
Trimio's compression savings analytics now use the corrected formula — compressed_tokens_in independent of caching semantics. The numbers in your dashboard are real. No 5× overstatement. No negative savings artifacts.