Writing the tests before the app, and letting an agent fix what fails overnight

I'm looking at a screen that's been finished for weeks and never actually tested. Not by me, not by anyone. My habit for a new screen has always looked the same: click through it myself, find what's broken, fix it, and once I know what "working" actually looks like, write the regression tests last, so nothing I just fixed quietly breaks again later.

I open the screen, get ready to start clicking, and stop. What if the tests come first this time, before I've looked at any of it myself, and an agent runs them against a screen neither of us has touched yet?

The order, flipped

Up to now, every screen in this build has gone through some version of the same order: a human drives it by hand first, an assistant captures what looks wrong, and an automated suite gets written afterward, once the bugs are already known, so they stay fixed. That order works. It also means the automated tests can only ever lock in what a person already found by clicking around.

This time the tests are how the bugs get found, not how they get remembered. Instead of exploring the screen, I open the test plan that already states, line by line, what this exact screen is supposed to do: what a returning visit should show, what an empty case looks like, what a specific action should do when it's launched from a specific record. Every line already has a stated correct answer. So instead of poking around to see what happens, I write one assertion per line, run it against the real screen, and treat every failure as a candidate bug, never as a test that needs adjusting to match whatever the screen currently happens to do.

Turning it loose overnight

Writing that first pass is the easy part. The harder question is what happens after. I'm not planning to sit and watch each assertion fail one at a time. An agent runs this loop by itself, overnight, while I'm asleep, with permission to change code. That part should make anyone nervous, and it made me nervous too, because the fastest way for any bug-fixing loop to reach a green run by morning is to bend the app until it agrees with whatever the test currently says, whether or not the test is the one that's actually wrong.

The rule that keeps it honest

So before a single line changes, every failing assertion gets sorted first. Is the test itself wrong, a bad locator, a stale assumption, in which case the test gets fixed and the app doesn't? Is it a small, deterministic style mistake with one obviously correct fix and no real judgment involved, in which case it just gets fixed and nobody needs to weigh in? Is the environment being unreliable right now, a slow cold start, a service still warming up, in which case the right move is to wait and retry and never treat it as a finding? Is there a genuine disagreement between what the screen does and what the test plan says it should do, confirmed by actually reproducing it, in which case a ticket gets opened and the app gets fixed? Or is there no clear right answer written down anywhere, a real design call with no obvious resolution, in which case it gets left alone and flagged, never guessed at?

Underneath all five of those is one rule. The test plan is the source of truth, not whatever the screen currently does. Reproduce the disagreement before changing anything. If it turns out reality already matches what was written down, the right move is to say so and move on, not to invent a fix for a problem that was never actually there.

What happened overnight

One screen, the one I'd been about to test by hand. Twenty-two tests, all authored from the test plan before a single one of them had ever run. Some assertions needed a version of the screen with its network calls faked, so I could check what renders without a real database sitting behind it. Others needed the opposite: a real database, because you can't prove what a server actually enforces by faking the server that's supposed to enforce it.

By morning: three real bugs, found and fixed. A field meant to show a readable label was instead showing the raw internal identifier behind it, because the query building the screen never looked the label up at all. A different field was supposed to reflect whoever a record's bill actually gets routed to, and it ignored that setting completely, always showing the record itself, because the read assembling the screen never checked it. And a button meant to open a form pre-filled for whatever record you'd clicked from instead opened to a blank search every time, because the form lives mounted once, high up in the app, and only reads its starting value the first time it mounts. Click that button for a specific record any time after the app has already been running, and the form has no way left to know what you wanted pre-filled.

One more line in the test plan flagged a suspect: a guard that was supposed to stop a record from being linked to itself. The agent went looking for that bug and didn't find one. The guard already existed, and it was already covered by an existing test, so nothing got filed. That's the outcome that's easiest to skip if the only thing being optimized for is a green run by morning, and it's the one I actually care about most, because it's proof the loop can tell a real bug apart from a hypothesis that just didn't pan out.

One small thing did slip past the line, and it's a good illustration of where that line actually sits. A label on the fixed screen had a punctuation mark that breaks a house rule for this app's copy. The agent noticed it, correctly judged it as a polish call rather than a functional one, and left it for me instead of guessing. Good instinct, wrong bucket: a fixed style rule isn't a judgment call, it's just something to check against. So that small miss became its own fix, and the fix became a standing addition to the rubric itself. A deterministic style rule gets corrected on the spot. It doesn't get deferred to a human who was never going to disagree with it anyway.

What still needed me

By the time I woke up, one pull request had already merged itself, the three real fixes bundled inside it, every check green. What a run like this proves is a functional contract: the screen does what the test plan says it should do. It says nothing about whether the screen looks right, whether the spacing feels correct, whether an interaction is actually pleasant to use. None of that is mechanically checkable, so the pass I still owed it by hand didn't disappear. It just got a lot shorter, because the only things left in it were things a test genuinely can't judge.

The plan becomes a tool

The playbook I hand-wrote for this one screen doesn't stay a one-off. It's already a tool now. Point it at any screen, and it reads the same kind of test plan, checks whether the right data is even seeded to make the assertions possible, decides which lines need a faked network and which need a real database, and hands back a run I can paste into an idle session and walk away from.

I didn't automate testing the screen. I automated writing the plan for testing the screen, and that turns out to be a different thing to hand off. It's also the part that was still mine to do, and the part I ended up keeping.

#BuildInPublic #ClaudeCode #AIAgenticEngineer #AIEngineering