Green CI, clean merge, and a schema that never reached production

I'm about to close the laptop when the production app starts throwing 500s. The patients list won't load. The error in the logs names a database role, app_runtime, and says it does not exist in production.

That doesn't make sense to me at first. I wrote the migration that creates that role. I merged it days ago. CI was green the whole way through, and the pull request merged clean. So how is the role missing from the one database that actually matters?

It takes a few minutes in the logs to find the answer, and the answer is worse than a bug. My deploy pipeline ships code on every merge and never runs migrations. The schema and the code have been drifting apart quietly for several merges. Tonight is just the night the drift finally had consequences.

The 500

The role app_runtime is not cosmetic. It came in with a batch of migrations that set up multi-tenancy and row-level security, and every runtime query opens with SET LOCAL ROLE app_runtime so the query runs as a restricted role that can only see one tenant's rows. If that role isn't in the database, every query fails on its very first statement. There is no partial degradation. The list just 500s.

Production had the early migrations, the ones that created the initial schema. It was missing the four newer ones that introduce the role, the row-level-security policies, an audit log, and a search index. On the throwaway databases where my tests run, all of them were present. On production, the last four had never been applied. The code that needed them had shipped anyway.

What the green check was actually checking

Here is the part that fooled me.

Every pull request spins up its own ephemeral database branch. That is a feature of the database provider I use, and the other managed Postgres and MySQL providers have their own version of it. CI creates a fresh branch, runs every migration against it from zero, runs the test suite, and then deletes the branch when the run finishes. This is genuinely useful. It proves the migration is syntactically valid, that it applies cleanly from a blank slate, and that the code passes its tests against the new schema. When CI went green, all of that was true.

What green CI never proved was that the migration reached production. It ran the migration against a database that exists for about ninety seconds and then evaporates. The production database was never in that loop. So the green check I had been reading as "this shipped" was answering a narrower question: "this migration is valid and the tests pass." Those are not the same sentence, and the space between them is exactly where four migrations went missing.

The deploy pipeline made it worse by being half-right. On every merge to the main branch it built the application and rolled it out. It just never ran pnpm db:migrate against the production database. Code shipped automatically; schema shipped never. For a few merges that gap did nothing visible, because the code didn't lean on the new schema yet. The merge that added SET LOCAL ROLE app_runtime to every query is the one that turned a silent gap into a wall of 500s.

Stopping the bleed

The first job is not to be clever. It is to make the app work again.

I write a small standalone workflow whose only purpose is to connect to the production database and apply whatever migrations it is missing. I trigger it by hand. It resolves the production connection, waits for the database to wake up, runs the migrations, and then does one more thing I add on purpose: it checks that the app_runtime role actually exists afterward and fails loudly if it doesn't. About four minutes after I hit run, the four missing migrations are applied and the patients list loads again.

That bought the app back. It did not fix the real problem, which is that nothing in the pipeline guarantees the schema is ever delivered. A button I have to remember to press is not a fix. It is the same gap with a longer fuse.

Making the schema a gated step

The permanent fix is small, and it is about ordering rather than cleverness.

I add a migration job to the deploy pipeline that runs before either of the deploy jobs, and I make the deploys depend on it. In GitHub Actions that dependency is a single line, needs:, but the behavior it buys is the whole point: if the migration fails, the deploys don't run. The pipeline would rather ship nothing than ship new code against a schema that didn't move. Migrate first, and only if that succeeds, roll out.

It keeps the safety check I added in the panic. After migrating, it asserts the role exists before the deploys are allowed to proceed. And I leave the manual workflow in place, not as the primary path anymore but as an escape hatch for the next incident, when I might need to migrate production without running a full deploy.

The fix itself was fast to carry out. An agent wrote the job in a background worktree as part of a larger batch of work that day, and the few minutes of typing were never the hard part. The hard part was the half hour before it, standing in the logs, working out that the question my CI had been answering was not the question I thought I was asking.

The check that wasn't checking what I thought

I trusted a green check. The check was honest. I was the one reading more into it than it said. "Valid and tested on a database that no longer exists" had quietly become "shipped" in my head, and the two drifted apart over several merges before a missing role made the difference loud.

What I keep turning over is that the pipeline is product code too. I had put real care into the schema, the tenancy model, the tests. I had put almost none into the system that delivers them. A delivery pipeline you don't design is still a design, just one you didn't choose on purpose. Schema delivery is now an explicit, ordered, gated step that blocks a rollout the moment it fails. It should have been one from the start. I just hadn't thought of the pipeline as something that deserved the same rigor as the thing it ships.

#BuildInPublic #CICD #DatabaseMigrations #DevOps #ClaudeCode