Point almost any coding assistant at a failing test and it will make it pass. That is the instinct the whole category is trained into, and for writing tests it is precisely the wrong one — because when an assertion fails, one of two things is wrong, and only one of them is the test.
The fork, and why the default answer is wrong
A red suite is easiest to fix by changing whichever side is not protected. Nothing is protecting the code — no test asserts that it stays as it is — so the cheapest repair is always to adjust the code until the assertion holds, or to soften the assertion until the code holds.
Do that once and the suite no longer says what the software should do. It says what the software happens to do, which is a thing you could have found out by running it.
So the rule is on the agent rather than left to the model's instincts: when a test it wrote fails, it works out which side is wrong, and if the answer is the code, it reports it and leaves the code alone.
The seeded file has two real bugs in it
Like every agent we build, this one opens on deliberately imperfect sample code — a small shopping cart with two genuine boundary bugs in it, not typos, but the kind of thing that survives review:
// Free shipping at 500 — or is it above 500?
function shippingFor(total) {
return total > 500 ? 0 : 49;
}
// Discount applies before or after tax, and the answer changes the total
function totalFor(items, discountPct) {
const subtotal = items.reduce((sum, i) => sum + i.price * i.qty, 0);
return subtotal * (1 - discountPct / 100) * 1.18;
}The first is an off-by-one at exactly the advertised threshold: a cart of exactly 500 pays shipping, which is not what the banner says. The second is an ordering question with a real answer in tax law and no answer in the code.
A clean sample would let the agent look competent while proving nothing. These two let it demonstrate the only behaviour that matters: finding a boundary, testing it, watching the assertion fail, and then not fixing the code.
It runs the tests. Really runs them.
A test that has never been executed is a wish. The agent has a Linux sandbox, installs what the project needs, and runs the suite:
$ npm test
✓ shippingFor > charges 49 below the threshold
✗ shippingFor > is free at exactly 500
expected 0, received 49
✓ totalFor > applies a percentage discount
1 failed, 2 passedThen it does the thing that makes it useful: it reads the failure, decides which side is wrong, and — for this one — tells you the code is. The test stays as written, red, with a note saying what the threshold appears to be intended as and what it actually is.
Where it puts the tests
Boundaries, first and mostly. Not one test per function, which is how you get a suite that is enormous, slow, and asserts nothing anyone doubted. The prompt pushes it toward:
- Exactly on the threshold, and one step either side. Three cases at every place behaviour changes.
- Empty, one, many. The three cardinalities where collection code goes wrong.
- The error path, asserted as a specific failure rather than “it throws”.
And it says what it did not cover. A coverage claim that implies completeness is worse than no claim, because it stops the next person from looking.
What it broke in our platform
Its first real run could not find npm. The sandbox's PATH had no /usr/local/bin in it, so node resolved and npm did not — which is why every check we had ever run passed. Nothing in the product had ever needed npm in a sandbox before, because people in a terminal type node.
That is the pattern behind the whole series: an agent is the first thing to walk a path end to end with nobody reading the middle, so it finds the parts that only fail when nobody is looking.
Use it on code you actually own
Paste in a module, or bring a repository, and ask for tests around a specific behaviour rather than for “tests”. The good prompt is a question: what happens at the free-shipping threshold, and is the code doing what the banner promises?