Serialization 03 📄 Schema Evolution, Backwards Compatibility, and Config Migrations
Configuration schemas change. A field is added, a key is renamed, a nested table becomes an array. The file on disk was written by an older version of the software, and the new version must either read it correctly or fail with a message the user can act on. Schema evolution is the discipline of managing those changes without breaking existing installations, and for YAML and TOML configuration the problem is harder than for databases because there is no server enforcing constraints—only the parser and the application code.
This chapter covers the compatibility taxonomy that determines which changes are safe, the expand-contract pattern that makes breaking changes deployable, the migration mechanisms that transform old files into new shapes, and the tooling that automates the process for YAML and TOML specifically. The sections that follow walk through backward-compatible changes, the expand-contract phases, and the migration implementation patterns that preserve user data and comments.
Key point: Adding optional fields is backward-compatible and requires no migration; renaming or removing fields is breaking and requires an expand-contract migration that reads old shapes, writes new shapes, and only removes the old path after every reader has moved.
Why schema evolution exists
The silent failure problem. A configuration file without a version field is interpreted as whatever shape the current code expects. When a field is renamed, the old file parses successfully but the field is ignored, and the application runs with a default value the user never chose. The failure is silent until the behavior diverges. Version fields and migration chains exist to make the divergence visible and correctable .
The comment preservation problem. TOML and YAML files are hand-edited. Comments explain why a value is set, and formatting makes the file scannable. A naive migration that deserializes the file, transforms the data structure, and re-serializes it loses every comment and every formatting choice. The toml_edit crate and similar libraries exist specifically to preserve comments and whitespace during programmatic edits, and migration frameworks that ignore this produce files the user no longer recognizes .
The multi-version problem. During a rolling deployment, old and new code run simultaneously. A config file written by the new code must be readable by the old code until the rollout completes, or the old instances crash on startup. The expand-contract pattern addresses this by making every change in two steps: first add the new structure alongside the old, then remove the old structure only after all readers have moved .
a. Backward-compatible versus breaking changes
The compatibility taxonomy is the foundation of every migration decision. A change is backward-compatible if old readers can still interpret the new file, or if new readers can interpret old files without data loss. It is breaking if either direction fails.
Adding a new optional field with a default is backward-compatible. The old reader ignores the field; the new reader supplies the default when the field is absent. Adding a new table is likewise compatible. Renaming a field, removing a field, changing a field’s type, or adding a required field without a default are breaking because the old reader cannot interpret the new shape or the new reader cannot find the data it needs .
| Change | Compatible | Reason |
|---|---|---|
| Add optional field with default | Yes | Old readers ignore; new readers default |
| Add new table | Yes | No existing path changes |
| Rename field | No | Old path disappears |
| Remove field | No | New reader loses data |
| Change type (string → int) | No | Parser may accept, semantics change |
| Add required field without default | No | Old files fail validation |
The version field in the configuration file is the mechanism that lets the application know which rules apply. A config_version field, checked on load, determines whether a migration is needed before the data is validated against the current schema . Without a version field, the application guesses, and the guess is wrong as soon as the first breaking change ships.
b. The expand-contract pattern for configuration
The expand-contract pattern, borrowed from database schema migration, makes breaking changes deployable without downtime. The pattern has three phases, each a separate deployment .
Expand: Add the new structure alongside the old. For a field rename, this means adding the new key while the old key still exists. For a type change, it means adding a new field with the new type and keeping the old field. Both old and new code can read the file; old code writes only the old key, new code writes both.
Migrate: Backfill the new structure from the old. For a field rename, copy the old key’s value to the new key where the new key is absent. For a configuration file, this is a migration function that runs on load and writes the file back in the new shape, or a one-time CLI command that transforms files in place.
Contract: Remove the old structure. Only after every reader has been updated to the new key can the old key be deleted. In a configuration context, this means the migration that removes the old key ships only after the release that stopped reading it.
The critical constraint is that each phase must be a separate release, and the contract phase cannot ship until the expand phase has been deployed long enough that every file in the wild has been migrated. For configuration files, this means the application must tolerate both shapes during the transition window, reading the new key first and falling back to the old key if the new one is absent.
c. Migration implementation for YAML and TOML
A migration is a function that receives the old configuration data and returns the new configuration data. The function must be idempotent (running it twice produces the same result), must preserve comments where possible, and must be testable independently of the application.
The fluxconf library for Pydantic-backed YAML configuration illustrates the pattern. Migrations are keyed by a string with an integer prefix that determines execution order, and they run automatically on read() . A migration can be a JSON Patch (RFC 6902) for simple structural edits or a Python function for conditional logic.
# JSON Patch migration: rename host to hostname
migrations = {
"1_rename_host": [
{"op": "move", "from": "/hostname", "path": "/host"}
],
"2_add_port": [
{"op": "add", "path": "/port", "value": 8080}
],
}For TOML, the migrable tool provides declarative migrations with comment-preserving edits via go-toml-edit, rollback through down operations, and a dry-run mode for previewing changes before writing . A migration file declares the operation, the path, and the default value.
# migrations/0001_add-debug.toml
description = "add debug flag to server section"
[[structure]]
op = "add_field"
path = "server.debug"
type = "boolean"
default = false
down = { op = "remove_field", path = "server.debug" }The version field in the configuration is updated after a successful migration, and the original file is backed up before any modification. These two rules prevent a failed migration from destroying the only copy of the user’s configuration and prevent the application from re-running migrations that have already been applied .
Complete Example Session
# ============================================
# PART 1: THE OLD CONFIGURATION (V1)
# ============================================
# config.yaml (v1 shape)
# version: 1
# server:
# hostname: localhost
# port: 8080
# ============================================
# PART 2: THE NEW CONFIGURATION (V2)
# ============================================
# config.yaml (v2 shape, after expand phase)
# version: 2
# server:
# host: localhost # new key
# hostname: localhost # old key kept for compatibility
# port: 8080
# ============================================
# PART 3: THE MIGRATION FUNCTION (PYTHON)
# ============================================
def migrate_v1_to_v2(data: dict) -> dict:
"""Copy hostname to host, keep both keys."""
server = data.get("server", {})
if "hostname" in server and "host" not in server:
server["host"] = server["hostname"]
data["version"] = 2
return data
# ============================================
# PART 4: THE LOAD PATH
# ============================================
import yaml
def load_config(path: str) -> dict:
with open(path) as f:
data = yaml.safe_load(f)
version = data.get("version", 1) # absent means v1
if version < 2:
data = migrate_v1_to_v2(data)
# write back the migrated file
with open(path, "w") as f:
yaml.dump(data, f, sort_keys=False)
return data
# ============================================
# PART 5: THE CONTRACT PHASE (LATER RELEASE)
# ============================================
# Remove hostname from the file and from the migration
# Only after every deployment reads host, not hostnameThe parts covered the old and new shapes during the expand phase, the migration function that copies the old key to the new key, the load path that runs the migration automatically, and the contract phase that removes the old key.
Quick Reference
Compatibility Taxonomy
| Change | Classification | Migration Required |
|---|---|---|
| Add optional field | Compatible | No |
| Add new table | Compatible | No |
| Rename field | Breaking | Yes (expand-contract) |
| Remove field | Breaking | Yes (contract only) |
| Change type | Breaking | Yes |
| Add required field | Breaking | Yes |
Expand-Contract Phases
| Phase | Action | Reader Behavior |
|---|---|---|
| Expand | Add new structure, keep old | Reads new first, falls back to old |
| Migrate | Backfill new from old | Reads new, ignores old |
| Contract | Remove old | Reads new only |
Migration Tooling
| Tool | Format | Key Feature |
|---|---|---|
fluxconf | YAML (Pydantic) | JSON Patch + function migrations |
migrable | TOML | Comment-preserving, rollback |
toml_edit | TOML | Preserves comments/formatting |
Best Practices
✅ Do This:
# Include a version field
version: 2 # ✅
# Keep old keys during expand phase
server:
host: localhost
hostname: localhost # ✅
# Back up before migrating
cp config.yaml config.yaml.bak # ✅
# Run migrations on load, not on write
if version < CURRENT: data = migrate(data) # ✅❌ Don’t Do This:
# No version field
server:
host: localhost # ❌ (cannot detect old files)
# Remove old key in the same release as the new key
server:
host: localhost # ❌ (old readers break)
# Deserialize and re-serialize without comment preservation
yaml.dump(yaml.safe_load(f)) # ❌ (loses comments)
# Run migration without idempotency check
data["version"] = 2 # no guard # ❌ (may re-run)Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Old config parses but field ignored | No version field or migration | Add version and check on load |
| Comments lost after migration | Re-serialization without preserving library | Use toml_edit or migrable |
| Old code crashes on new file | Contract phase shipped too early | Wait one deployment cycle |
| Migration runs twice | No idempotency guard | Check version before transforming |
| Required field breaks old files | Added without default | Add optional with default first |
Real-World Examples
1. Version field in YAML
version: 22. Version field in TOML
config_version = "2.0.0"3. JSON Patch migration (fluxconf)
{"op": "move", "from": "/hostname", "path": "/host"}4. Declarative TOML migration (migrable)
[[structure]]
op = "add_field"
path = "server.debug"
default = false5. Comment-preserving edit (toml_edit)
# toml_edit loads and writes without losing comments6. Backup before migration
cp config.yaml config.yaml.bak7. Idempotent migration guard
if data.get("version", 1) < 2:
data = migrate(data)8. Fallback read during expand
host = server.get("host") or server.get("hostname")9. Contract after rollout
# Remove hostname only after all readers use host10. Dry-run preview
migrable migrate --dry-runVisual
Compatibility Spectrum
┌─────────────────────────────────────────────────────────┐
│ CHANGE TYPES │
│ │
│ COMPATIBLE BREAKING │
│ ────────── ──────── │
│ Add optional field Rename field │
│ Add new table Remove field │
│ Widen type Change type │
│ Add required field │
│ │
│ No migration Migration required │
│ Deploy anytime Expand-contract │
└─────────────────────────────────────────────────────────┘Expand-Contract Timeline
┌─────────────────────────────────────────────────────────┐
│ THREE RELEASES │
│ │
│ Release N: EXPAND │
│ ┌──────────────────────────────────────────┐ │
│ │ Old key: present │ │
│ │ New key: added, written by new code │ │
│ │ Readers: read new, fall back to old │ │
│ └──────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Release N+1: MIGRATE │
│ ┌──────────────────────────────────────────┐ │
│ │ Migration backfills new from old │ │
│ │ All files in the wild are now v2 │ │
│ └──────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Release N+2: CONTRACT │
│ ┌──────────────────────────────────────────┐ │
│ │ Old key removed │ │
│ │ Readers: new only │ │
│ └──────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘Migration Load Path
┌─────────────────────────────────────────────────────────┐
│ ON LOAD │
│ │
│ Read file │
│ │ │
│ ▼ │
│ Check version field │
│ │ │
│ ┌────┴────┐ │
│ ▼ ▼ │
│ Current Old │
│ │ │ │
│ ▼ ▼ │
│ Validate Migrate │
│ │ │
│ ▼ │
│ Write back │
│ │ │
│ ▼ │
│ Validate │
└─────────────────────────────────────────────────────────┘Comment Preservation
┌─────────────────────────────────────────────────────────┐
│ NAIVE MIGRATION PRESERVING MIGRATION │
│ │
│ parse → dict toml_edit / migrable │
│ │ │ │
│ ▼ ▼ │
│ transform in-place edit │
│ │ │ │
│ ▼ ▼ │
│ serialize write back │
│ │ │ │
│ ▼ ▼ │
│ Comments LOST Comments KEPT │
└─────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Compatible changes | Add optional field, add table, widen type |
| Breaking changes | Rename, remove, type change, required without default |
| Expand phase | Add new alongside old, readers fall back |
| Migrate phase | Backfill new from old |
| Contract phase | Remove old after all readers updated |
| Version field | Required to detect old files |
| Comment preservation | toml_edit, migrable |
| Backup rule | Copy before any migration |
Key takeaways:
- The compatibility taxonomy decides everything. Adding an optional field is safe; renaming or removing a field is not. The classification determines whether a migration is needed and which pattern applies .
- Expand-contract makes breaking changes deployable. Add the new structure first, backfill it, and remove the old structure only after every reader has moved. Each phase is a separate release .
- A version field is the entry point. Without it, the application cannot know whether a file needs migration, and the failure mode is silent: the old field is ignored and the default takes over.
- Comment preservation is not optional for hand-edited configs. A migration that loses comments produces a file the user no longer recognizes.
toml_editandmigrableexist to prevent this . - Migration must be idempotent and backed up. Running the same migration twice must produce the same result, and the original file must be copied before modification so a failed migration does not destroy the only copy.
Remember: Configuration schema evolution is the same problem as database schema evolution, without the database. There is no server to enforce constraints and no transaction log to roll back. The version field is the only signal, the migration function is the only transformation, and the expand-contract timeline is the only safe path for breaking changes. When a field must change its name, add the new name first, keep the old name readable, and remove it only after the last deployment that reads it has shipped.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!