The day your schema change breaks a dashboard nobody asked about is the day you understand what schema evolution actually is.
The three phases are not symmetrical
A schema change has three phases. The first is the easiest to get wrong because it feels complete.
Phase one: write
You add the column. You ship the writer. You run the migration. The test suite passes. This is when people say "we're done." They are not.
The data is in the new shape. Half the system knows it. The other half reads the old shape and waits for an error that will not come because databases tolerate unknown fields gracefully. That tolerance is what makes the gap invisible until a consumer casts a value that is no longer there.
Phase two: read
You deploy the reader. It works on the new shape. It produces empty values on the old shape. This is the longest phase and also the most dangerous because it creates the illusion of progress. Nothing is broken yet, but the system is running a mixed-shape workload, and the mixed shape is where bugs hide.
A consumer that reads the old field first and the new field second will behave differently depending on which version it talks to. A cache populated with the old shape will not reflect the new one until it expires. A query that filters on the old column will stop returning results. A report that joins on the old structure will produce incomplete data without warning.
The work here is not "turn on the reader." It is making sure every path through the system handles both shapes without changing behavior from the consumer's perspective. The API layer needs to normalize between old and new. The cache needs to be cleared or versioned. The queries need to handle both columns. The reports need to use the same logic regardless of shape. The documentation needs to reflect the change. The on-call runbook needs to mention the new field. That is coordination work. It is not DDL.
Phase three: delete
You remove the old field. You remove the old code paths. You clean up the migrations. This is when people finally remember that someone else might still be reading.
A third-party integration polling your API once a day does not see the deployment window. A data warehouse snapshotting the table on Sunday morning still has the old column for another week. A batch job on a different schedule is still writing the old shape for a few more hours.
The delete phase is when you apologize.
Backward compatibility is not a library
A schema registry tells you whether a new schema breaks compatibility with an old one. It does not tell you whether every consumer has upgraded, whether the data warehouse is still reading the old column, or whether a third-party integration is still sending the old field name.
Compatibility enforcement is a policy decision, not a technical one. The technical side can check whether a new field is added or removed, whether a type changes, whether a required column is dropped. The policy side has to ask: is this consumer ready to see this change? Is this API version still in use? Has the data team updated their queries?
Those are coordination questions. The tooling can surface them. It cannot answer them.
The people who handle schema evolution well do not have better tools than the people who handle it poorly. They add fields before removing them. They write code that handles both shapes simultaneously. They deploy the writer before the reader, the reader before the old code, and the old code removal last. They check every known consumer before deleting anything and keep a list of known consumers that they update when new ones appear.
That list is the single most underrated artifact in schema evolution. It is rarely maintained. It is always out of date. It is the first thing people look for when something breaks.
The dependency graph nobody maintains
The real shape of the problem is a dependency graph. You know this graph exists because you have seen it when a change breaks something unexpected.
flowchart TD
DB[(Database)] --> API[API Service]
API --> FE[Frontend]
API --> MOBI[Mobile App]
API --> ETI[ETL Jobs]
API --> BI[BI Dashboards]
API --> AUTH[Auth Service]
ETI --> DW[Data Warehouse]
DW --> ANALYTICS[Analytics]
BI --> PRODUCT[Product Team]
Every arrow is a potential apology. When you change the schema, you change the contract that the database exposes. Every service that reads from it is a node in that graph. Every downstream consumer of those services is another node. Every dashboard, report, or third-party tool that depends on any of those nodes is another arrow.
The graph grows quietly. New integrations appear. Old services get forgotten. Data warehouse snapshots accumulate. Mobile apps ship slowly and roll back partially. The graph is always bigger than the team thinks it is.
Most teams discover the size of the graph on the day something breaks.
The teams that handle schema evolution well maintain a smaller version of this graph in their heads. They know which services are fast to update and which are slow. They know which consumers are internal and which are external. They know which data paths are synchronous and which are asynchronous. They use that knowledge to order their deployment steps.
The order matters. Writing the new shape before reading it is obvious. Reading it before deleting the old shape is also obvious. The non-obvious part is ordering the deployment of consumers. The team that knows the mobile app takes six weeks to roll out will not delete a field that the mobile app uses before that rollout completes. The team that does not know has to learn through an apology.
The postmortem pattern
Schema evolution postmortems follow a recognizable pattern. The root cause is never "the database was not updated." It is that the team changed the schema without checking if any consumer was still using it. It is that they assumed a consumer had upgraded when it had not. It is that they deployed the reader before the writer, or removed the old field before a batch job had finished running. It is that they did not have a list of known consumers, or tested in staging while production data was already in a mixed shape, or thought backward compatibility meant "the database accepts both shapes" when it actually means "every consumer handles both shapes."
None of these are surprising. They are all predictable. They are all prevented by the same discipline: treat the schema as a contract between producers and consumers, not as a definition owned by one team.
The contract analogy is useful because contracts have two sides. The writer promises to produce data in a certain shape. The reader promises to handle that shape. When the shape changes, both sides have to renegotiate. The side that stops listening is the side that ends up apologizing.
The one-line version
Schema evolution is not a DDL problem — it is a coordination problem between producers and consumers, and the DDL is just the moment you realize how big the coordination gap actually is.
Previous: Architecture Is What Happens After Success
Next: Recovery Is the System