Training data is a moat. Documentation is how you rent it back.

I've been measuring what an AI coding agent costs on Guren compared to five other frameworks. The numbers turned out to be less about the framework than about the model, and it took a few rounds of re-measuring to see why.

The setup: give Claude Code the same task, add a complete tagging feature to the same blog app, on six implementations in this repo. Schema, forms, display, ?tag= filtering, validation, tests. Three trials each, headless. Scoring is blind and automated. Each trial starts from a fresh worktree that is verified green, and after the session the harness applies the patch to another fresh worktree, re-runs typecheck and the full test suite, and then runs a functional smoke the agent never saw: seed SQLite directly, boot the app, verify ?tag= filtering over real HTTP.

Measured bare, in July 2026, Guren cost $5.54 median against hono's $2.03, nearly 2.7 times as much.

The gap was archaeology, not friction

My assumption was that Guren made agents work harder: more layers, more ceremony, more places to get lost.

The session streams told a different story: the implementation work itself was fast and error free. What consumed the budget was reading:

trial	tool actions	actions in node_modules/@guren/*	share
guren-2	101	18	17%
guren-3	103	48	46%
guren-4	139	60	43%

Between 17% and 46% of everything the agent did was opening compiled bundles and .d.ts files under node_modules/@guren/* to work out what the API actually looked like. Agents on hono and Next.js touched node_modules close to zero times. They didn't need to: the model already knew those APIs.

The cost wasn't Guren being hard to use. It was Guren being unknown, and the agent paying, in tokens, to rediscover it from build output.

Fixing docs moved the number, then stopped

So I wrote guidance and re-measured. Rounds 2 through 4, July 2026: a CLAUDE.md that auto-loads, then a leaner one plus .claude/rules/*.md carrying exact, source-verified signatures. The median went 5.54, then 4.51, then 3.35. Turns went 104, 95, 77.

I nearly published that as a 40% improvement, but before doing so I re-ran the whole thing on current releases, because the framework had moved a lot and I didn't want to ship a number that only held for a snapshot. It didn't.

guren median	bare	shipped harness	+ signature digest
Cost (USD)	6.26	6.08	4.99
Turns	94	102	81

The harness as actually released was delivering −3%. The win from round 4 had quietly evaporated between the experiment and the product.

Attach-on-edit is pull with extra steps

The streams explained it. Round 4's win came from agents reading the rule files. The shipped harness delivered those same rules as glob-scoped rules that auto-attach when a matching file is edited. But roughly 75% of the node_modules archaeology happens before the first edit. By the time the rules attached, the agent had already paid for the answers they contained.

Two related findings from the same streams:

• Skills went unused. Files the agent has to decide to invoke were used zero to one times across every run, in every round.
• The CLAUDE.md pointer to the rules was ignored. Naming a resource is not the same as providing it.

Push beats pull, and attach-on-edit turns out to be pull with extra steps.

The fix was to stop asking. Guren's guren context command already runs at session start through the harness's hook, so the API signature digest moved inside that output. Signatures now arrive before any work starts. First edit moved from around tool 46 to around 34, pre-edit archaeology dropped from 23–24 actions to about 10, and the arm landed at −20% cost, −14% turns against bare.

The control that changed nothing

The result I trust the most here is the one where my own change did nothing.

Next.js ships its own agent guidance. I stripped it, measured, restored it, and measured again. The difference was ±0.

That's the actual finding. Guidance pays off where the model lacks API knowledge; where it already knows the framework from training data, more documentation buys nothing, because the agent was never going to go looking in the first place.

Training-data familiarity is a real, measurable advantage for established frameworks. Documentation engineering is how a young framework rents it back.

When guidance stops being an optimization

Guren v2.0.0 shipped breaking majors. I re-ran both arms the next day, when claude-sonnet-5 had zero training data for the new API shapes.

	bare	shipped harness
Acceptance	1/3	3/3
Cost (USD)	6.94	4.90
Turns	93	93

The number that matters here is acceptance, more than the −29% cost. Both bare failures produced a feature that passed the hidden HTTP smoke, so the thing worked, and then failed the test gate. The agents had guessed at v2 API shapes from stale training data, gotten them wrong, and written tests around the wrong guesses, confidently, in code that looks fine in a diff.

That's the failure mode I care about now. Models mostly stopped producing obviously broken code a while ago, but what they produce on an API they haven't seen is plausible code with wrong assumptions baked into the tests that were supposed to catch it.

With the harness, all three trials shipped. On a major the model has never seen, guidance stops being an optimization and becomes the difference between shipping and not.

Lifetime acceptance across all rounds is 43 of 45. The only two failures are stripped-guidance arms on a day-old major.

What I would not claim

• One model, one task, N=3. The medians are stable enough to locate an effect, not to size it precisely. Per-trial spreads overlap.
• Absolute costs are not comparable across rounds. The host environment changed between July and August. Arms within a round are comparable to each other, and that is all.
• The July cost table is a July snapshot, on the framework versions of that date. The 40% arc from rounds 2 to 4 did not survive contact with a heavier environment. −20% with the digest, and −29% on the v2 day-one round, are the honest numbers.
• Frameworks the model knows have a structural advantage no scaffold fully erases. That advantage is part of what I measured, not something I controlled away.

Every round is published, including the two where my own changes delivered nothing: the round-5 −3% and the Next.js ±0. The harness, the task definition, the raw session streams, and the lab notes are all in the repo.

agent-eval · PILOT.md (round by round) · STUMBLES.md (stream analysis) · TASK.md (judge the task yourself)

Why this shaped the framework

Guren ships the result of all this: every scaffolded app gets an agent harness, and guren context hands over a project map with API signatures at session start rather than waiting to be asked. guren check and guren audit exist because the other half of the problem is verification, and plausible-wrong code is exactly what a human reviewer is worst at catching.

None of it came out of a design meeting; it came from reading session streams and noticing that a third of the agent's budget was going to archaeology.