Serialization 12 📄 Apache Avro — Schema Embedding and Dynamic Typing - KandZ

⏱️ Estimated read: 10 min | |

Apache Avro is a data serialization system built on a different premise than Protobuf or MessagePack: the schema is always present when data is read, and it travels with the data rather than being compiled into the reader. This design choice produces Avro’s defining properties—dynamic typing, untagged data, and symbolic field resolution—and it is the reason Avro became the serialization layer for Hadoop, Kafka, and data pipelines that need to read historical data written by older versions of the schema.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

This chapter covers Avro’s schema model, the embedding mechanisms that carry the writer’s schema to the reader, and the resolution rules that allow a reader’s schema to differ from the writer’s. The sections that follow walk through the schema definition, the writer/reader resolution algorithm, and the schema evolution rules that determine which changes are safe.

Key point: Avro data is always accompanied by the writer’s schema; the reader uses that schema to decode the bytes and an optional reader’s schema to resolve differences, which is why Avro supports dynamic typing without code generation and why schema evolution is a first-class feature rather than an afterthought.

a. Schema definition and the JSON representation

An Avro schema is defined in JSON. The primitive types are null, boolean, int, long, float, double, bytes, and string. Complex types are record, enum, array, map, union, and fixed . A record has a name, a namespace, and an ordered list of fields, each with a name, a type, and an optional default. The field order matters for serialization: a record is serialized as the concatenation of its field values in declaration order, with no field names or tags on the wire .

{
  "type": "record",
  "name": "User",
  "namespace": "com.example",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "email", "type": "string"},
    {"name": "age", "type": ["null", "int"], "default": null}
  ]
}

A union is represented as a JSON array of schemas. Avro’s union has specific rules: it may not contain more than one schema with the same type (except named types), and it may not immediately contain other unions . A union is serialized by writing a long index that indicates which branch was used, followed by the value encoded per that branch’s schema . The ["null", "int"] union with default: null is the canonical way to represent an optional field, because the default must match the first branch of the union for backward compatibility .

b. Schema embedding and writer/reader resolution

Avro’s schema is not compiled into the reader; it is carried with the data. In a container file, DataFileWriter prefixes the file with the schema used to write all the records, and DataFileReader reads that prefix to deserialize the data correctly . In RPC, the client and server exchange schemas during the connection handshake, and since both sides have the other’s full schema, differences in field names, missing fields, and extra fields can be resolved symbolically . When records are stored outside a container file—in a database or a message queue—the writer’s schema must be stored or referenced alongside the record, typically by a schema registry that assigns an ID .

The resolution algorithm is applied when the reader’s schema differs from the writer’s. The specification defines the matching rules: both schemas must be the same primitive type, both arrays with matching item types, both maps with matching value types, both enums with matching names, both records with the same name, or the writer’s type must be promotable to the reader’s (int → long → float → double) . If both are records, fields are matched by name, not position; the ordering may differ, and schemas for fields with the same name are resolved recursively . If the writer has a field the reader lacks, the writer’s value is ignored. If the reader has a field the writer lacks, the reader uses its default value if one exists, or signals an error if none does . This is the mechanism that makes schema evolution work: the reader can be a newer version of the schema that added fields with defaults, and the old data resolves cleanly.

c. Schema evolution and compatibility rules

Avro’s compatibility is defined by what the reader can resolve. The safe changes are adding a field with a default value, removing a field that had a default, changing documentation or aliases, and changing a non-union type to a union containing only the original type (or vice versa) . The unsafe changes are renaming a field without an alias, changing a field’s type, and adding a field without a default when the reader expects backward compatibility .

Change	Backward Compatible	Reason
Add field with default	Yes	Reader uses default for old data
Add field without default	No	Old data has no value, no default to use
Remove field with default	Yes	Writer’s value ignored by reader
Rename field without alias	No	Field matching is by name
Change type (int → string)	No	No promotion rule applies

The compatibility mode determines the upgrade order. BACKWARD (the default in schema registries) means new consumers can read old data, so consumers are upgraded first. FORWARD means old consumers can read new data, so producers are upgraded first. FULL means both directions work, and the upgrade order is flexible . The transitive variants check against all previous versions rather than just the latest, which matters when consumers might replay from the beginning of a topic .

Avro’s performance position is modest against the protocol-based formats. A study comparing serialization protocols found that Avro showed a slight edge over JSON regarding throughput, but the advantage was less significant than MessagePack’s, and Protobuf achieved the highest throughput for the smallest data sizes . For the largest record size, Avro presented the lowest median latency at 2.71 ms, compared to JSON’s 4.18 ms . The performance case for Avro is not raw speed; it is the schema evolution guarantee and the dynamic typing that avoids code generation for generic data-processing systems.

Complete Example Session

// ============================================
// PART 1: THE WRITER'S SCHEMA (V1)
// ============================================
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "email", "type": "string"}
  ]
}

// ============================================
// PART 2: THE READER'S SCHEMA (V2, ADDED FIELD WITH DEFAULT)
// ============================================
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "email", "type": "string"},
    {"name": "country", "type": "string", "default": "US"}
  ]
}

// ============================================
// PART 3: RESOLUTION
// ============================================
// Old data written with V1 has no "country" field.
// The reader uses V2's default "US" for that field.
// The resolution succeeds; no error.

// ============================================
// PART 4: THE UNSAFE CHANGE
// ============================================
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "email", "type": "string"},
    {"name": "country", "type": "string"}  // no default
  ]
}
// Old data has no "country" and no default exists.
// The reader signals an error: "Reader missing default value"

// ============================================
// PART 5: UNION FOR OPTIONAL FIELDS
// ============================================
{"name": "age", "type": ["null", "int"], "default": null}
// The default matches the first branch ("null")
// Old data resolves to null for "age"

The parts covered the writer’s schema, the reader’s schema with a defaulted field, the successful resolution, the unsafe addition without a default, and the union pattern for optional fields.

Quick Reference

Schema Types

Category	Types
Primitive	`null`, `boolean`, `int`, `long`, `float`, `double`, `bytes`, `string`
Complex	`record`, `enum`, `array`, `map`, `union`, `fixed`
Union	JSON array of schemas, no duplicate types
Optional field	`["null", "type"]` with `default: null`

Resolution Rules

Writer	Reader	Result
`int`	`long`, `float`, `double`	Promoted
Record field in writer, not reader	—	Ignored
Reader field with default, absent in writer	—	Default used
Reader field without default, absent in writer	—	Error
Enum symbol absent in reader	—	Error

Compatibility Modes

Mode	New Consumers Read Old	Old Consumers Read New
BACKWARD (default)	Yes	No
FORWARD	No	Yes
FULL	Yes	Yes
NONE	No checking	No checking

Best Practices

✅ Do This:

// Always add defaults to new fields
{"name": "country", "type": "string", "default": "US"}       // ✅

// Use union with null for optional fields
{"name": "age", "type": ["null", "int"], "default": null}     // ✅

// Test compatibility before registering
// curl .../compatibility/...?verbose=true                    // ✅

// Use aliases for renamed fields
{"name": "email", "aliases": ["email_address"]}              // ✅

❌ Don’t Do This:

// Add a field without a default (BACKWARD mode)
{"name": "country", "type": "string"}                        // ❌

// Rename a field without an alias
{"name": "email2"} // was "email"                             // ❌

// Change a field's type
{"name": "id", "type": "string"} // was "long"                // ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Deserialization error on old data	New field lacks default	Add `default` to the field
Field silently missing	Writer field not in reader	Expected; reader ignores it
Rename breaks consumers	No alias defined	Add `aliases` to the field
Schema registry rejects registration	Compatibility mode violated	Test with compatibility API

Real-World Examples

1. Record schema

{"type": "record", "name": "User", "fields": [...]}

2. Optional field union

{"name": "age", "type": ["null", "int"], "default": null}

3. Defaulted field

{"name": "country", "type": "string", "default": "US"}

4. Enum schema

{"type": "enum", "name": "Status", "symbols": ["A", "B"]}

5. Array schema

{"type": "array", "items": "string"}

6. Schema in container file

DataFileWriter prefixes the file with the schema

7. Compatibility test

curl .../compatibility/subjects/user-value/versions/latest

8. Field alias

{"name": "email", "aliases": ["email_address"]}

9. Promotable type

int → long, float, double

10. Union index on wire

long index + value per branch

Visual

Schema Embedding

┌─────────────────────────────────────────────────────────┐
│  AVRO CONTAINER FILE                                    │
│  ┌──────────────────────────────────────────┐           │
│  │ Schema (JSON)                            │           │
│  │ { "type": "record", ... }                │           │
│  ├──────────────────────────────────────────┤           │
│  │ Record 1 (binary, no field names)        │           │
│  │ Record 2 (binary, no field names)        │           │
│  └──────────────────────────────────────────┘           │
│                                                         │
│  The schema travels with the data                       │
│  The reader uses it to decode                           │
└─────────────────────────────────────────────────────────┘

Writer/Reader Resolution

┌─────────────────────────────────────────────────────────┐
│  WRITER SCHEMA (V1)        READER SCHEMA (V2)           │
│  { id, email }             { id, email, country }       │
│         │                        │                      │
│         ▼                        ▼                      │
│  Old record: { id: 1, email: "a@b.com" }                │
│         │                                               │
│         ▼                                               │
│  Reader resolves: country = default ("US")              │
│         │                                               │
│         ▼                                               │
│  Output: { id: 1, email: "a@b.com", country: "US" }     │
└─────────────────────────────────────────────────────────┘

Compatibility Modes

┌─────────────────────────────────────────────────────────┐
│  BACKWARD (default)                                     │
│  New consumers read old data → consumers first          │
│                                                         │
│  FORWARD                                                │
│  Old consumers read new data → producers first          │
│                                                         │
│  FULL                                                   │
│  Both directions → any order                            │
└─────────────────────────────────────────────────────────┘

The Default Rule

┌─────────────────────────────────────────────────────────┐
│  ADD FIELD                                              │
│  ┌──────────────────────────────────────────┐           │
│  │ with default → old data resolves          │           │
│  │ without default → error on old data       │           │
│  └──────────────────────────────────────────┘           │
│                                                         │
│  REMOVE FIELD                                           │
│  ┌──────────────────────────────────────────┐           │
│  │ with default → safe (reader ignores)      │           │
│  │ without default → still safe (ignored)    │           │
│  └──────────────────────────────────────────┘           │
└─────────────────────────────────────────────────────────┘

Summary

Item	Value
Schema format	JSON
Primitive types	`null`, `boolean`, `int`, `long`, `float`, `double`, `bytes`, `string`
Complex types	`record`, `enum`, `array`, `map`, `union`, `fixed`
Schema embedding	Container file prefix, RPC handshake, registry ID
Field matching	By name, not position
Promotion	`int` → `long`/`float`/`double`; `long` → `float`/`double`
Safe addition	Field with default
Unsafe addition	Field without default (BACKWARD)
Compatibility default	BACKWARD

Key takeaways:

Avro’s schema travels with the data. In a container file, the writer’s schema is prefixed to the file; in RPC, it is exchanged in the handshake; in a registry, it is referenced by ID. The reader always has the schema it needs to decode .
Dynamic typing means no code generation is required. The schema is data, not compiled into the reader, which is why Avro works well for generic data-processing systems and dynamic languages .
Field matching is by name, not position. Record fields can be reordered, and the resolution still works. This is the symbolic resolution that Avro’s design enables .
Adding a field with a default is safe; without a default it is not. The reader uses the default for old data that lacks the field. Without a default, the reader has no value to supply and signals an error .
Avro’s performance is modest; its evolution guarantee is the reason to choose it. It showed a slight edge over JSON in throughput and the lowest latency for the largest record size in one study, but Protobuf and MessagePack are faster for small data .

Remember: Avro is the serialization format for data that outlives the code that wrote it. The schema is not a compile-time artifact; it is a runtime companion that travels with every file and every RPC handshake. That is why schema evolution works: the reader compares the writer’s schema to its own and resolves the differences symbolically, using field names and defaults. Add fields with defaults, use unions for optional values, and test compatibility before registering. The format’s performance is not its selling point; its ability to read yesterday’s data with today’s code is.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!