We are building one agent a day on Mume AI, in public, and publishing each one as we go. Five are live. Every single one of them has found a bug in our own platform that nothing else had found — not the test suite, not the review, not the fact that we use this thing all day.
That is not a coincidence and it is not bad luck. It is the actual reason the series is worth the time, and it is the argument of this post: an agent is the first thing that exercises a path end to end. A person using a chat app stops at every step to read what came back. An agent does not. It sends the request, gets the response, and acts on it, twenty times in a row, with nobody watching the middle.
So it finds the parts of your system that only fail when nobody is looking.
The rule, before the stories
Every bug below has the same shape: a thing that was correct at both ends and never connected in the middle. Both ends type-check. Both ends have tests. The wire between them is not a thing you can write a test for, because in the test the wire is a mock.
We now have a rule about it, in our own contributing notes: a field declared at both ends is not evidence. Go and look at the wire.
Agent 01 — CSV Analyst, and the system prompt nobody sent
The first agent reads a spreadsheet, profiles it before it counts anything, and writes charts as real files. It has a long, carefully written system prompt — the whole personality of the thing, the rule about checking data before summarising it, the instruction to name what it could not read.
None of it was ever sent to the model.
useEffect(() => {
setSystemPrompt("");
}, []);That ran on mount. It was there to clear a stale prompt between sessions, which is a real problem it really solved. It also discarded the initial value that the app config had just supplied, one tick after supplying it.
The reason it survived so long is the interesting half: no surface had ever had a system prompt to lose. Chat does not set one. The editor does not set one. CSV Analyst was the first thing in the product with an opinion to send, and so it was the first thing that could notice the opinion was being thrown away.
The model, meanwhile, did a perfectly reasonable job without it. That is what made it invisible: the failure mode of a missing system prompt is not an error, it is a slightly more generic answer.
Agent 02 — Repo Explainer, refused by our own rate limiter
The second agent clones a public repository, reads it, and writes an architecture note where every claim carries the file path it came from. Reading a repository properly means a lot of small calls: list, open, open, search, open.
Our API rate limit was 60 requests a minute. One worst-case turn of this agent is about 75.
We had written that limit for someone abusing the platform. The first client to ever hit it was us, doing exactly what the product is for. It is a useful reminder that a limit is not a security control, it is ashape — and if you have never measured the shape of your own heaviest legitimate client, you have picked a number out of the air and called it a policy.
Agent 03 — Test Writer, and the missing bin directory
The third agent writes tests for your code and then runs them, which is the whole point of it: a test that has never been executed is a wish. It reports the failures rather than quietly rewriting your code until the suite goes green.
Its first real run could not find npm.
$ npm test
sh: 1: npm: not found
$ node --version
v22.11.0The sandbox's PATH had no /usr/local/bin in it. Node itself resolved, because Node is somewhere else. So every check we had ever done — “is Node in the image? yes” — passed, and the one thing an agent actually needs to run a JavaScript test suite was missing.
Nobody had noticed because until Test Writer, nothing in the product had ever run npm in a sandbox. Humans in the terminal type node.
Agent 12 — Voice-over, and every media tool we had
This one is the big one, and it is the reason we started writing these down.
Voice-over transcribes a recording, translates it, and speaks it back in another voice — three model surfaces chained in a single run. On its first attempt, the speech step failed. So did image generation. So did every other media tool an agent could call.
They had been broken since the day they were written. Not regressed — never once worked.
The cause is a difference between two processes running identical code. Our media routes resolve a storage bucket from the environment. In a Cloud Function, Firebase populates FIREBASE_CONFIG and the bucket resolves. The media studio in the web app calls the Cloud Function, so the studio worked, and it worked in front of us every day.
Agents reach the same routes over the Cloud Run loopback instead. Cloud Run sets no FIREBASE_CONFIG — and, we discovered on the second attempt, no GOOGLE_CLOUD_PROJECT either.
// Worked in a Cloud Function. Silently undefined on Cloud Run.
const bucket = JSON.parse(process.env.FIREBASE_CONFIG ?? "{}").storageBucket;The part worth telling on ourselves: the first fix deployed cleanly and failed in exactly the same way. We had swapped one absent variable for another absent variable, watched a green deploy, and believed it. The second fix names the bucket explicitly, because the environment had now lied to us twice.
Same code. Same request. Different process. That is a category of bug that no unit test reaches, and it stayed hidden for as long as the only caller was the one process where it happened to work.
Agent 04 — Meeting to Tasks, and a gate that was not there
The newest one turns a wall of meeting notes into issues you can assign, and files them into Notion or Linear. It is the first agent we have built that writes into somebody else's workspace, which is why it was next: it is the first real test of the approval gate, the thing that stops a model from acting without you.
The gate did not cover it.
needsApproval("writeFile", "manual"); // true
needsApproval("linear_create_issue", "manual"); // falseThe function resolves a tool against our platform registry to decide whether it writes. A tool from a connected MCP server is not in that registry, so it fell through to “no approval needed”. The result was precisely backwards: a run in manual mode would stop and ask before writing a file inside its own sandbox, and then create issues in a real team's tracker without a word.
The fix was already sitting there in the protocol. MCP servers send an annotations.readOnlyHint with each tool, saying whether it only reads. We were parsing it and throwing it away. Now we keep it, a write parks the same way any other gated tool does, and a missing hint is treated as a write — silence from a third party should read as “might write”, not as permission.
Then the same agent found the follow-up. With the gate fixed, the agent's own approvalMode: "manual" policy was still being ignored — because the composer sends an approval mode on every request, and the server could not tell “the user chose automatic” from “the control starts at automatic”. A correct definition, a correct resolver, and a default in between quietly winning. The same shape as everything above.
What we changed about how we build them
Two rules came out of this that we now apply to every new agent.
Seed it with deliberately imperfect data. The CSV Analyst's sample spreadsheet has mixed date formats, a number stored as text, a duplicate row and a blank in a numeric column. The Test Writer's sample code has two real boundary bugs. The Meeting to Tasks notes contain a contradiction nobody resolved, a decision reversed four lines later, and an item that was already done. A clean sample lets an agent look competent while proving nothing.
Give every agent its own opening. Every route used to open on “How can I help you today?”, which is the sentence a general chat app says because it does not know. An agent does know, and the first screen is where it should say so.
Why not just use the product more
We do use it. Every day. It found none of these.
Using a product exercises the paths a person is patient enough to walk. Building an agent on it exercises the paths where nobody is patient at all — the retry, the twentieth tool call, the write that happens while you are reading the previous answer. Those are the paths where a missing environment variable, an over-tight limit or an absent gate lives, and they do not surface until something automated leans on them.
There is a version of this where you build agents around a platform: an integration layer, a wrapper, a separate service that calls your API from outside. It would have found none of these either, because it would have brought its own environment with it.
Twenty-five to go. We expect every one of them to break something, and we will write down what.
The five that exist so far are on the agents page — CSV Analyst, Repo Explainer, Test Writer, Voice-over and Meeting to Tasks. Each has its own page saying what it does, how it is set up, and what it will not do.