About half of what passed in dev didn’t survive staging. A feature would work perfectly on someone’s machine, get merged, reach staging, and fall over. The code was fine. The database was lying to everyone.
Here’s how the lie worked. We had one shared development database. Everyone connected to it, and everyone could edit it directly through a GUI. Need a new column to test something? Add it by hand. Tweak a type? Right click, change it. It was fast, and it felt productive.
The problem is that the schema now lived in two places that disagreed: the migration files in the repo, and the actual running database that people had been quietly editing for months. Dev “worked” because someone had hand added the column your feature needed. Staging was built from the migrations, which never heard about that column. So staging broke, the author swore it worked on their machine, and they were telling the truth.
Ours had quietly stopped being a source of truth and become a source of drift. Every manual edit is an undocumented migration that exists in exactly one place and survives only until someone runs a reset.
The fix was less about tooling and more about a rule: humans do not write to the database. Only automation does.
Concretely, for staging:
- the database was locked down, so people get read access, not write. No more hand edits.
- the schema became declarative migrations in a single repo, with one tool (Atlas) owning the entire history.
- migrations run as a pre deploy step, so what reaches staging is exactly the migration history and nothing a human snuck in on the side.
The moment a human can poke the schema directly, it stops being something you can reason about. Lock it, and the only way to change the shape of the database is a migration in version control that everyone can see and CI can replay. Staging stops lying because there is nothing left to lie about: dev, staging, and prod are all built from the same files.
There is an obvious objection, and it is the real reason people hand edit in the first place. They need realistic data and a realistic schema locally, and an empty migrated database gives them neither. So the other half of the fix is giving each developer their own throwaway database, seeded with a masked, realistic slice of production, that they can break freely. Once our local databases were realistic and disposable, the pull to hand edit a shared one went away.
The thing I keep coming back to: once the schema lived somewhere a human could quietly change it, it drifted, and that drift surfaced in the one place I least wanted surprises, right before prod.