Serialization 07 📄 Introduction to Binary Serialization

⏱️ Estimated read: 9 min | |

Binary serialization converts structured data into a sequence of bytes that is not human-readable but is compact and fast for machines to process. It is not a single format; it is a family of formats—Protobuf, Avro, MessagePack, CBOR, FlatBuffers—that share the property of encoding data without the field names, punctuation, and decimal digits that text formats carry. The choice to use binary is driven by volume: at high message rates, the encoding cost dominates, and the byte savings compound; at low rates, the difference is noise and the tooling cost outweighs the benefit .

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

This chapter covers the mechanics that make binary formats smaller and faster, the schema-based versus schema-less distinction that determines how they evolve, and the decision rules that map message rate and consumer type to format choice. The sections that follow walk through the encoding mechanics, the schema distinction, and the conditions under which binary serialization earns its complexity.

Key point: Binary serialization trades human readability for compactness and speed; it is the correct choice when the consumer is a machine and the message volume is high enough that encode/decode cost matters more than debuggability.

Why binary serialization exists

The field-name problem. JSON repeats every field name in every message. A record with call_id, talkgroup, and duration_seconds pays for those strings every time it is serialized. Protobuf encodes the same data as numeric tags and values; the field names exist only in the .proto schema, not in the payload. The measured difference is roughly 4× for structured records: the same call record is 107 bytes in JSON and about 25 bytes in Protobuf . A worked example with a three-field message shows 41 bytes in JSON and 12 bytes in Protobuf—a 70.7% reduction .

The parsing problem. Text formats require the parser to scan for delimiters, convert decimal digits to integers, and handle quoting and escaping. Binary formats with length-prefixed fields can jump directly to the bytes they need. A benchmark of distributed communication formats found that Avro, Protobuf, and Thrift reduced median latency by 80–84% compared to JSON . The IEEE study confirms that binary and protocol-encoded formats benefit from increased batch size because they are optimized for compactness and machine efficiency, while text-based protocols experience reduced throughput due to verbose nature and increased parsing complexity .

The compression caveat. Gzipped JSON often lands within about 2× of Protobuf’s size, which is the standard rejoinder to binary adoption. But compression adds CPU on both ends, adds latency per message, and does nothing for parsing cost—after decompression, the parser still parses text . Compression narrows the size gap, not the parsing gap. It is a fine tool for shrinking a text API’s bandwidth; it is not a refutation of binary formats where encode/decode rate is the constraint.

a. Schema-based versus schema-less binary formats

Binary formats divide into two families, and the distinction determines how the format evolves and what tooling it requires.

Schema-based formats (Protobuf, Avro, Thrift) require a schema definition outside the payload. The schema provides field names, types, and—in Protobuf—stable field numbers that enable forward and backward compatibility. A reader that does not recognize a field number skips it; adding a new field is safe by construction, and renaming a field is free because the name is compile-time only . The cost is that the schema must be distributed and versioned, and debugging requires schema-aware tooling.

Schema-less formats (MessagePack, CBOR, BSON) encode data without a separate schema. They are self-describing in the sense that types are embedded in the payload, but they do not enforce a contract. MessagePack and CBOR are compact and fast, but they do not provide the schema evolution guarantees of Protobuf or Avro. They are appropriate when the data structure is simple, stable, or already agreed upon out of band .

The schema question is the evolution question. A schema-based format with a registry (Avro + Schema Registry, Protobuf + .proto versioning) can evolve safely across producers and consumers. A schema-less format leaves evolution to application discipline .

b. When binary serialization earns its complexity

The decision is volume-driven. The encoding matters in proportion to message rate × message count × lifetime .

Scenario	Format	Reason
Config fetch at startup	JSON/YAML/TOML	Microseconds, optimize for eyes
Dashboard API at 100 req/s	JSON	Overhead small vs network latency
Internal RPC at high rate	Protobuf/gRPC	Size and parse cost dominate
Audio chunks at 50/s × hours	Binary	Megabytes/hour, constant parse load
Millions of msgs/s between services	Binary	Encoding is the dominant cost

The rule of thumb is that if the message rate is low and the message lifetime is short, the encoding is noise and debuggability wins. If the message rate is high and the messages persist, the encoding is the product and binary wins . The IEEE study adds a nuance: binary formats benefit more from batching than text formats, so a workload that can batch is a stronger candidate for binary .

The tooling cost is real and must be accounted for. Binary traffic is debuggable only through schema-aware tooling—grpcurl, Protobuf decoders, protocol dissectors—that must be present, current, and pointed at the right schema. That is a maintained dependency, and in a small project it is often the first thing that rots . A project that adopts Protobuf without adopting the debugging tooling has traded debuggability for size without building the replacement.

Complete Example Session

# ============================================
# PART 1: THE SAME DATA IN TWO ENCODINGS
# ============================================
# JSON (41 bytes):
# {"id":42,"name":"widget","in_stock":true}

# Protobuf (12 bytes):
# 08 2a 12 06 77 69 64 67 65 74 18 01
# Field 1 (id): tag 0x08, value 42 (varint)
# Field 2 (name): tag 0x12, length 6, "widget"
# Field 3 (in_stock): tag 0x18, value 1 (true)

# ============================================
# PART 2: THE VOLUME DECISION
# ============================================
# Startup config:   JSON overhead ~80 bytes, irrelevant
# 100 req/s API:    JSON overhead small vs network, irrelevant
# 50 chunks/s audio:  megabytes/hour, binary earns its keep
# Millions msg/s:   binary, no debate

# ============================================
# PART 3: SCHEMA-BASED VS SCHEMA-LESS
# ============================================
# Protobuf: .proto schema required, field numbers stable
#   Adding a field: safe (reader skips unknown)
#   Renaming a field: safe (name not on wire)
#   Reusing a field number: corrupts data

# MessagePack: no schema, self-describing types
#   Fast and compact, but no evolution guarantees

# ============================================
# PART 4: THE TOOLING COST
# ============================================
# Binary debugging requires:
#   grpcurl (for gRPC)
#   protoc --decode (for raw Protobuf)
#   Schema registry (for Avro)
# These are maintained dependencies, not optional

The parts covered the encoding difference, the volume decision, the schema distinction, and the tooling cost.

Quick Reference

Binary Format Families

Format	Schema	Evolution	Debug Tooling
Protobuf	Required (.proto)	Field numbers, skip unknown	protoc, grpcurl
Avro	Required (embedded)	Schema resolution	Avro tools, registry
MessagePack	None	Application discipline	Generic decoders
CBOR	None	Application discipline	Generic decoders
FlatBuffers	Required	Field offsets	Schema-aware

Decision Rules

Condition	Choice
Low message rate, short lifetime	Text
High message rate, persistent	Binary
Human consumer	Text
Machine consumer with schema	Binary
Batching workload	Binary benefits more

Size Comparison (1 KB object)

Format	Encoded Size	Relative
JSON	1000 B	Baseline
JSON+gzip	400 B	2.5× smaller
Protobuf	300 B	3.3× smaller
Avro	280 B	3.6× smaller

Best Practices

✅ Do This:

# Use Protobuf for internal high-rate RPC
service UserService { rpc Get(Req) returns (Resp); }         # ✅

# Adopt debugging tooling with the format
# grpcurl, protoc --decode, schema registry                  # ✅

# Measure before switching
# baseline latency, payload size, parse CPU                  # ✅

# Use schema registry for Avro/Protobuf evolution
# (versioned schemas, compatibility checks)                  # ✅

❌ Don’t Do This:

# Use Protobuf for a startup config fetch
# (adds tooling for no measurable gain)                      # ❌

# Adopt binary without debugging tooling
# (opaque payloads at 2 a.m.)                                # ❌

# Reuse a Protobuf field number
# (silently corrupts data)                                   # ❌ 

# Assume compression solves the parsing gap
# (size yes, parse no)                                       # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
JSON at high message rate	Familiarity, not measurement	Measure encode/decode cost
Protobuf without tooling	Size win, debuggability loss	Adopt `grpcurl` or decoder
Schema drift in Protobuf	Field number reused	Reserve old numbers forever
Expecting compression to fix parsing	Size and parse are separate costs	Binary only if parse cost matters
Schema-less format evolves badly	No contract enforcement	Use schema-based format with registry

Real-World Examples

1. Protobuf message definition

message Call {
  int32 call_id = 1;
  int32 talkgroup = 2;
  float duration_seconds = 3;
  bool encrypted = 4;
}

2. Protobuf encoding

08 95 f8 02 10 b1 09 19 ...  # ~25 bytes

3. JSON equivalent

{"call_id":48213,"talkgroup":1201,"duration_seconds":8.4}

4. MessagePack (schema-less)

Compact binary, no schema, fast encode/decode

5. Avro with schema registry

Schema stored externally, evolution checked

6. gRPC service

service CallService { rpc GetCall(GetCallRequest) returns (Call); }

7. Debugging with protoc

protoc --decode=Call call.proto < payload.bin

8. Schema evolution rule

reserved 5;  // never reuse field number 5

9. Batching benefit

Binary throughput improves with batch size

10. Hybrid architecture

Public: JSON. Internal: Protobuf. Config: YAML.

Visual

The Encoding Difference

┌─────────────────────────────────────────────────────────┐
│  JSON (41 bytes)                                        │
│  {"id":42,"name":"widget","in_stock":true}              │
│                                                         │
│  Protobuf (12 bytes)                                    │
│  08 2a 12 06 77 69 64 67 65 74 18 01                    │
│                                                         │
│  Field names not on wire                                │
│  Numbers as varints                                     │
│  ~70% size reduction                                    │
└─────────────────────────────────────────────────────────┘

Schema-Based vs Schema-Less

┌─────────────────────────────────────────────────────────┐
│  SCHEMA-BASED                    SCHEMA-LESS            │
│                                                         │
│  Protobuf, Avro, Thrift          MessagePack, CBOR      │
│                                                         │
│  Field names in schema           Types in payload       │
│  Field numbers stable            No evolution contract  │
│  Skip unknown fields             Reader must agree      │
│  Registry recommended            Application discipline │
│                                                         │
│  Evolution guaranteed            Evolution manual       │
└─────────────────────────────────────────────────────────┘

Volume Decision

┌─────────────────────────────────────────────────────────┐
│  MESSAGE RATE × COUNT × LIFETIME                        │
│                                                         │
│  Low volume                                             │
│  ┌──────────────────────────────────────────┐           │
│  │ Config fetch, dashboard API              │           │
│  │ → Text wins, debuggability matters       │           │
│  └──────────────────────────────────────────┘           │
│                                                         │
│  High volume                                            │
│  ┌──────────────────────────────────────────┐           │
│  │ Audio chunks, service mesh RPC           │           │
│  │ → Binary wins, encoding dominates        │           │
│  └──────────────────────────────────────────┘           │
└─────────────────────────────────────────────────────────┘

The Tooling Cost

┌─────────────────────────────────────────────────────────┐
│  BINARY DEBUGGING REQUIREMENTS                          │
│                                                         │
│  Without tooling:                                       │
│  ┌──────────────────────────────────────────┐           │
│  │ 08 95 f8 02 10 b1 09 19 ...              │           │
│  │ (opaque bytes)                           │           │
│  └──────────────────────────────────────────┘           │
│                                                         │
│  With tooling:                                          │
│  ┌──────────────────────────────────────────┐           │
│  │ call_id: 48213                           │           │
│  │ talkgroup: 1201                          │           │
│  │ duration_seconds: 8.4                    │           │
│  └──────────────────────────────────────────┘           │
│                                                         │
│  Tooling must be present, current, and schema-aware     │
└─────────────────────────────────────────────────────────┘

Summary

Item	Value
Size reduction (JSON → Protobuf)	~70% for small records
Latency reduction (schema binary vs JSON)	80–84%
Schema-based formats	Protobuf, Avro, Thrift
Schema-less formats	MessagePack, CBOR, BSON
Evolution mechanism (Protobuf)	Stable field numbers, skip unknown
Tooling requirement	Schema-aware decoders
Batching benefit	Binary > text
Decision driver	Rate × count × lifetime

Key takeaways:

Binary formats trade readability for compactness and speed. The same record is roughly 4× smaller in Protobuf than JSON, and parsing is faster because there are no delimiters to scan or decimal digits to convert .
Schema-based formats provide evolution guarantees; schema-less formats do not. Protobuf’s field numbers allow adding and renaming fields safely; MessagePack and CBOR leave evolution to application discipline .
Compression narrows size, not parsing cost. Gzipped JSON is smaller on the wire, but after decompression the parser still parses text .
The decision is volume-driven. At low rates, the encoding is noise and debuggability wins. At high rates, the encoding is the product and binary wins .
Tooling is a maintained dependency. Binary debugging requires schema-aware tools that must be present and current. Adopting binary without the tooling trades debuggability for size without building the replacement.

Remember: Binary serialization is not better than text; it is different. It spends tooling to save bytes and CPU. The question is whether the bytes and CPU matter at the volume the system runs. A config file read once at startup should be readable by a human. A stream of millions of records between services should be compact and fast. Measure the volume, identify the consumer, and choose the format the consumer expects—not the one that wins a benchmark at the wrong scale.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!