I tried to build a "token optimization stack" for coding agents. Here's why I killed it.
A pilot benchmark, a $5.60 receipt, and the 97%-savings number that turned out to be a silent failure.
How it started
This started from a plain, unglamorous problem: I kept running into token limits at work. I was using Claude Code for real engineering work, burning through context and budget faster than I wanted to, and the obvious next question was — can I actually cut that down without cutting into the quality of what the agent produces? Not "use a cheaper model and hope," but something more deliberate: a stack of tools that each attack a different part of the token bill — how much context gets read, how much gets re-read, how verbose the agent's own output is, how it navigates a codebase in the first place.
That question turned into a side project: token-optimization-stack, a public repo documenting a composable set of tools meant to sit around Claude Code and reduce token spend, plus token-stack-benchmarks, a benchmarking harness to actually measure whether any of it worked.
I'm writing this up because the project ended, a few weeks in, in a place I didn't expect: not with a validated stack and a savings number, but with a concrete demonstration that the token-savings numbers I was looking at were actively misleading, and a cost model that made the whole exercise stop making sense before I even got to publish a result. I think both of those are more useful to write about than a clean win would have been.
What the stack looked like early on
The first version of the stack had five tools in it:
- Graphify — turns a codebase into a queryable knowledge graph, so the agent can ask "what calls this" instead of grepping and reading files to find out.
- Serena — symbol-level code navigation and editing via LSP, instead of raw file reads and text edits.
- Headroom — advertised as transparent context compression that "requires no behavioral changes" once installed.
- LiteLLM — a routing layer, with the idea being complexity-based model routing: cheap model for simple subtasks, expensive model only when needed.
- Caveman — output-side compression; a terser interaction style plus compressed subagent output, to cut down on verbose back-and-forth.
Two of those five didn't survive contact with an actual benchmark harness.
Headroom turned out not to do what its own documentation implied. The only registration path that didn't involve wrapping the whole claude invocation in a separate launcher (headroom init claude) registers an on-demand MCP tool — something the agent can call, not something that transparently compresses context whether the agent calls it or not. headroom doctor confirmed nothing was actually being routed through it without a separately running proxy process and an ANTHROPIC_BASE_URL override — a materially heavier integration than "requires no behavioral changes" suggested. On top of that, its mcp serve subcommand crashed outright against a current MCP SDK install, needing an old, pinned mcp<2 dependency to even start.
LiteLLM had a more fundamental problem: its usage-based routing is a load-balancing strategy across provider endpoints, not the complexity-based, per-subtask routing the idea in my head required. And Claude Code, mechanically, sends one fixed model for an entire -p session — there's no hook to swap models mid-task based on subtask complexity. The tool I wanted didn't exist yet in the shape I needed it.
Both got removed rather than kept in as unverified claims. What was left — Graphify, Serena, a context-compression/caching layer (LeanCTX), and Caveman — became the actual stack under test.
Why I had to stop
Not because the idea was wrong — that came later. The experiment itself stopped being economically sane to keep running.
The rigorous version of this — SWE-bench Verified and Multi-SWE-bench tasks, sixteen repos, five ablation arms, three repeats each — works out to roughly 4,800 agent runs. I never got near that. A deliberately cheap pilot instead: 31 tasks, 2 arms, one repeat, on the cheapest model I had (claude-haiku-4-5, medium effort). It partially completed — 11 of 31 pairs — at roughly $5.60 in raw API cost alone, before EC2 compute, Docker builds, or the multi-day slog of getting a Linux-shaped harness running on both EC2 and an Apple Silicon Mac.
Extrapolate that to the full matrix and you're past $1,200 in API spend, on the cheapest model available, before a single result is trustworthy. Sonnet is priced roughly 3x Haiku on both input and output tokens (published per-token rates — not accounting for whatever token-count difference a stronger model would actually produce), so ruling it out on cost alone would put the same matrix past $3,600. And haiku wasn't trustworthy: zero correct fixes on the Java-track tasks, and the full stack broke two of the three Python tasks the plain baseline had already solved.
Is this savings number — or this correctness failure — actually about the stack? Or is it about the fact that I'm running everything on the cheapest model I could afford to run 4,800 times of?
I didn't have a good answer. That ambiguity, at that price, is where I stopped.
What I actually found, for what it's worth
Even the partial pilot data was worth publishing, because it directly contradicts what a token-savings-only view would have told me. On the three Python-track tasks the plain baseline agent solved correctly, adding the full optimization stack:
- On one, the agent used the stack's terser operating style to treat a fully self-contained bug report as ambiguous, asked a one-line clarifying question on turn one, and stopped —
num_turns: 1, no error, nothing left to score. The plain baseline took 30 turns on the identical prompt and fixed it. Read purely off the token dashboard, this run posted 97% fewer tokens used — the single best-looking number in the whole pilot, produced by the one run that did no work at all. - On a second, a real patch was produced, it applied cleanly, and the target test still failed.
- The third stayed correct, but used more tokens than baseline, not fewer.
I'm not treating "2 of 3" as a rate — three tasks isn't a sample you generalize a percentage from. What does hold, even at this size, is the shape of the failure: token metrics and correctness metrics pointed in opposite directions, and the worst outcome in the batch produced the best-looking number. That doesn't need a bigger n to be true — it happened, on a real task, and it's exactly the failure mode a token-savings-only report can't see.
I'd also expect this to get worse, not better, on the model I used. Haiku is the model least equipped to recover when a terser operating style leaves it less room to push back usefully or reason through whether a task is actually ambiguous — a stronger model plausibly asks the same question but keeps working anyway, or doesn't need to ask at all. I didn't get to test that; it's a specific, checkable prediction for whoever picks this up next, not just a shrug.
Token and cost savings numbers, without a real correctness check against the actual test suite, aren't just incomplete — they can point in exactly the wrong direction. And the biggest, flashiest savings number is a plausible place for that to happen, not an unlikely one.
The fix is a one-line change to how anyone should report this kind of number: report cost per solved task, not cost per task. Under that metric the 97%-savings run isn't a win with an asterisk — it's infinitely expensive, because the denominator is zero. That single substitution closes the trap; a dashboard built around it can't launder a silent failure into a headline number.
If you want to pick this up
I'm not going to keep running this — not because I think the underlying question is answered, but because I can't afford to answer it properly on my own budget with the current tooling. If someone wants to take this further, both repos are public:
- token-optimization-stack — the stack itself: setup scripts and docs for Graphify, Serena, LeanCTX, and Caveman, plus the benchmarking methodology this whole pilot followed.
- token-stack-benchmarks — the harness: Dockerized ablation-arm runner, task sampling, and the SWE-bench / Multi-SWE-bench scoring scripts, including everything I found along the way getting a Linux-shaped benchmark harness to run correctly on both EC2 and an Apple Silicon Mac — case-sensitivity bugs, architecture mismatches, a brand-new Python version breaking a scoring dependency, and more.
Contributions, forks, and "actually here's why your pilot was wrong" pull requests are genuinely welcome. A few concrete open threads if you want a starting point:
- Why does the full stack produce empty or broken patches on tasks the plain baseline solves correctly? Not yet root-caused. Could be the compression layer dropping context the agent actually needed, could be something else entirely.
- The full 5-arm ablation was never run — only a 2-arm baseline-vs-full-stack pilot. It's entirely possible one specific tool in the stack is responsible for the regression and the others are fine, or even helping.
- Nobody has run this on a stronger model. Everything above is haiku at medium effort, specifically because it was the only model cheap enough to pilot at all. It's an open question whether Sonnet or Opus, at real per-run cost, would show a real positive effect that a weak model can't demonstrate — I just couldn't afford to find out.