The bug is fixed. Now I have to prove it stays fixed.

I just fixed a bug. The manual QA pass caught it: I was driving a screen by hand, an AI partner was watching and taking notes on what broke, and one behavior was wrong. The fix took a few minutes. Then I sat there looking at it with a much less comfortable question. What stops this exact thing from breaking again the next time I touch this screen?

Two different questions wearing one name

The previous post in this series covered how the test plan gets built: read every acceptance criterion in the specs, turn it into a checklist, then run that checklist screen by screen with an AI partner capturing what goes wrong in real time. That pass answers one question. Is this screen doing the right thing, right now?

That question needs a human in the loop. Judging whether a screen's behavior matches the intent behind it, the messy, non-literal kind of correctness, isn't something I hand off. The AI partner drives the note-taking and the triage; I drive the clicking and the judging.

But "right now" is doing a lot of work in that sentence. The moment I move to the next screen, or touch a shared component, or bump a dependency three layers down, the guarantee expires. The second question, does this screen keep doing the right thing, is a completely different job. It doesn't need judgment. It needs memory. That's what an automated suite is actually for, and it's the layer I started building once the manual passes started producing fixes worth protecting. I write it in Playwright, and once the manual pass has already decided what "correct" looks like, an AI agent can write most of the actual check itself. What it can't do is skip the manual pass and decide correctness on its own.

Making the sign-in cheap

The straightforward way to write an automated check for a screen behind a login is to sign in at the start of every single test. I tried that first. Every test opened the real sign-in screen, typed a real email and password, waited for a one-time code, and only then started checking anything. It worked, and it was slow: a handful of seconds of pure login overhead multiplied by every test in the file.

It also broke in a way that looked like flakiness but wasn't. The authentication provider throttles repeated sign-ins from the same account inside a short window, the same protection that stops a bot from hammering a login form. Running the suite a few times back to back tripped that throttle, and a test would fail on the sign-in step itself, nothing to do with the screen it was supposed to be testing. It read exactly like a flaky test. It was a rate limit.

The fix was to stop signing in per test. One dedicated step signs in once, for real, and saves the resulting session. Every other test loads that saved session directly and starts already authenticated. One login gates the whole run instead of one login per test, and the throttle stopped being a factor because there was only ever one real sign-in to trigger it.

When the test is the one lying to you

Two failure modes cost me more debugging time than any actual app bug, and both came from the test lying about what it found.

The first: a patient's name doesn't appear once on a screen. It's in the breadcrumb, the page header, a row in a table, sometimes a receipt line, all at once. Ask the test to find "the element with this name" and it finds four of them, refuses to guess which one you meant, and fails before your actual assertion even runs. Numbers have the mirror-image version of the same problem: a loose match on a short value like "5" can silently match inside a much larger one like "1,500" on the same screen. The fix, both directions, is to stop asking "find this text anywhere" and start saying "look inside this specific section, then find this text," or matching on a value specific enough that only one element could hold it.

The second is quieter and more dangerous. I group related checks together so a screen's whole flow runs as one ordered sequence, and by default that group stops at the first failure and skips the rest. A report that says "1 failed, 11 skipped" reads like a small, contained problem. It might be eleven unknowns wearing a "skipped" label instead of a real result. I stopped treating a skip as a formality. Every skipped line gets read as a genuine gap until I've re-run it and watched it pass.

The safety net I couldn't afford to run on every change

The obvious plan, once the suite existed, was to run all of it before every single change reaches the app. I did that for a while. It cost ten to fifteen minutes per change and kept growing as more screens got covered.

My first instinct was the one everyone reaches for: run it in one browser instead of four. It saved about thirty-five seconds. Barely worth mentioning next to a ten-minute run.

The real cost was never the browser matrix. It was the sheer number of tests that require a signed-in session, close to a hundred and twenty of them by the time I measured, each one paying for a real render of a real authenticated app rather than for an extra browser. Scoping the browser doesn't touch that number at all.

So the full Playwright suite came off the per-change path entirely. It runs once overnight, on its own schedule, and once more, in full, immediately before anything reaches the real deployment, so a regression surfaces within a day at worst and never reaches production. Every change still gets checked before it merges, just by faster, cheaper layers in between: does it build and typecheck, do the isolated unit checks pass, and, for anything touching the API, does it hold up against a real, disposable database spun up and torn down just for that one change. The expensive click-through-the-whole-app layer is reserved for the two moments that actually need to see the whole app behave, not for every line changed along the way.

The manual pass is the layer that decides what's correct. The automated one is the layer that remembers. I didn't expect, back when the manual pass was still turning up the bug that started this, that half the work of building the second layer would end up being about a schedule and a budget instead of the tests themselves. But that's where deciding where the safety net actually runs, not just that it exists, turned out to live.

#BuildInPublic #Playwright #ClaudeCode #AIAgenticEngineer