If Agents Write the Code, Who Says What Good Is?

Agents are collapsing the distance between the person who knows what a piece of software should do and the software itself. That is the encouraging half of the story, and I made that case on its own. This is the part that comes right after, and it is less comfortable.

If you want an agent to run without a human checking every step, closing the distance is not enough. You also have to hand it a way to tell whether it got there: not just the need, but how you know the need is met, including the parts we rarely write down. Quality, performance, security, and the plain question of whether the thing does what it was for.

Saying what “good” means is not separate from directing the agent. Past a certain point it is how you direct it. A goal with no way to check it just sends the agent back to ask you, again and again.

You move the work, you do not remove it

So agentic development does not remove the hard work. It moves it. The effort that used to go into writing the code now goes into two things: saying what good means, and building a way to measure the gap between what the agent produced and that. Both were always there. We hid the first inside specifications nobody could fully write, and the second inside a human reading the code. Take the human out of the middle and both come due at once.

There is a familiar shape for this, and it is the OKR. You give the agent an objective, which is the need, and key results, which are how you will know the need is met. Hand it an objective with no key results and you have handed it an aspiration. It will produce something, and you will have no way to tell whether it is the thing you wanted, short of looking yourself, which is the manual review you were trying to get out of.

The catch is the one every team that has ever written OKRs already knows. The key results are the hard part. For performance you can write one: a latency target. For security you can borrow one: a scan for known vulnerability classes. For “does it answer the need,” you often cannot write it down at all, and that inability is the whole difficulty, not a detail beside it.

There is no free oracle for “answers the need”

Software testing has a name for this. It is called the oracle problem: given some input, deciding whether the system did the right thing is itself hard, and for some programs there is no practical way to do it (Barr et al., “The Oracle Problem in Software Testing,” 2015). Most of the checks that matter are oracle problems wearing a disguise.

I used to think performance and security were a different kind of thing from “answers the need,” one machine-checkable and the other a matter of taste. I have come around to thinking it is a difference of degree. “Answers the need” could be machine-checked too, if you could state it precisely enough. You usually cannot, because stating it precisely means getting a picture out of one person’s head and through an organization without it degrading on the way.

So the key results split in two. Some you buy off the shelf. The decisive one you have to manufacture, per context, by encoding what your business actually means by good. That is what a company’s knowledge is for. It is not only there to answer questions; it is the rubric the agent grades itself against, and the quality of that rubric sets the ceiling on how far the agent can go alone. Put plainly: the autonomy you can hand an agent is the fraction of “done” you can express as a check it can run for itself.

The people at the frontier are saying the same thing

This is not only my read. When OpenAI described how it works with its own coding agents, it put the surviving human job like this: humans “translate user feedback into acceptance criteria, and validate outcomes” (Harness engineering). Anthropic, writing about how to build agents, frames evaluations as giving a team “a clear hill to climb,” turning “the agent feels worse” into something you can act on (Demystifying evals for AI agents). Both are pointing at the compass, not the engine.

There is a useful piece of honesty in the numbers, too. A controlled study by METR in 2025 had experienced developers work on mature codebases they knew well, and found that AI tools made them about 19% slower, even while they believed the tools had sped them up (METR, 2025). The study is careful about its scope, and it is not a verdict on AI coding in general. But the gap between felt and measured is the whole point. Steer by how it feels and you steer wrong. You need the check.

Two ways this goes wrong

The first is Goodhart’s law, in its popular phrasing: when a measure becomes a target, it stops being a good measure. An agent is the most literal target-chaser you will ever manage. Point it at a key result that does not fully capture the need and it will hit the number and miss the point, at machine speed. The check is only as trustworthy as the intent baked into it, which puts you straight back at the knowledge problem.

The second is assuming human judgment goes away. It moves. It stops being spent reading every diff and starts being spent owning the checks that decide when the work is done. Anthropic still calls human review crucial at the level of whether a solution fits the wider system. Armin Ronacher, the engineer behind the Flask framework, put it bluntly: testing and evals are the hardest problem in agent design, not the part you get for free.

The corner where it all lines up

When the pieces do line up, the result is startling. The runtime Bun was rewritten from one language to another, more than a million lines, by agents in eleven days, by its team’s own account (Rewriting Bun in Rust). None of it was magic. A human steered. A fleet of agents ran in parallel. The budget was, for practical purposes, unlimited: the tokens were counted, but cost was never the thing that stopped anyone. And there was a test suite precise enough to say when the job was done. The open budget mattered because it let the agents keep trying against that suite until they passed it. With nothing to try against, the same budget would only have produced more unreviewed code, faster.

But read the timeline again. The eleven days got the agents to a passing test suite. Shipping took months more. And Bun’s test suite is unusually thorough, one of the best in its class. If it still took months to go from “tests pass” to “safe to ship” with an oracle that good, that gap is the measure of how much “good” the tests did not yet capture. Passing the check and being ready were not the same thing, and the distance between them was the part of “good” nobody had written down.

That corner is worth staring at. If you always have a reliable way to check “done,” and tokens are effectively free, technical debt starts to lose its meaning. Debt is the interest you pay later for a shortcut taken now. But if an agent can regenerate the code to hit the check whenever the check changes, the shortcut and the clean version cost about the same to reach, and the thing that made debt expensive, the human hours to pay it down, is exactly the thing that just got cheap. How far that actually holds is an argument for another day. The condition it rests on is the same one this whole piece is about: a reliable way to say what good is.

Where good cannot be written down

There is a floor under all of this. A task can be handed to an agent to the exact degree that its “good” can be written down. Where “good” resists being written down, the human stays. The philosopher Michael Polanyi called this the tacit dimension: we know more than we can tell. You recognize good taste, a clean design, the right tone for a hard message, long before you can hand anyone the rule for it.

Two honest caveats, so this does not become a comfort blanket. First, that line moves. What we cannot express today we often learn to express tomorrow, or we stop trying to write the rule and let the agent learn it from our judgments instead, which is what training on human preference already is. Where you cannot specify good, you can still be the one who recognizes it, and that turns the human from the author of the work into the source of the signal. Second, the work that stays human is the work that is hard to measure, which is precisely the work an organization is quickest to underinvest in, because it carries no number.

So the end of programming, if that is what this turns out to be, does not end the hard part. It moves it into the open. It is also why I have stopped treating “coding” and “engineering” as the same word. Coding is producing the code, and that is the part the agents are taking. Engineering is knowing what good means, saying it clearly enough that something else can build toward it, then checking that it did. The first was never the hard part. The second was always the job.