Serialization 09 📄 Protocol Buffers — Field Numbers, Wire Format, and Compatibility Rules - KandZ

⏱️ Estimated read: 9 min | |

Protocol Buffers’ efficiency and evolution guarantees both rest on one design decision: the wire format identifies fields by number, not by name. The field number is packed into a tag alongside a three-bit wire type, and that tag is what the parser reads to decide how to decode the following bytes . The field name exists only in the .proto schema and the generated code; it never appears in the payload. This is why renaming a field is safe on the binary wire and why reusing a field number corrupts data silently.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

This chapter covers the wire format’s tag structure, the wire types that determine how values are encoded, and the compatibility rules that classify schema changes as safe, risky, or breaking. It explains why int32 is inefficient for negative numbers, how the one-byte versus two-byte tag ranges work, and the deletion pattern that reserves a field number forever. The sections that follow walk through the tag encoding, the wire type mapping, and the compatibility taxonomy.

Key point: The tag is (field_number << 3) | wire_type, encoded as a varint; field numbers 1–15 cost one byte, and a deleted field’s number must be reserved so it is never reused.

a. The tag and the wire types

Every field in a protobuf message is encoded as a key-value pair. The key is the tag: a single integer that packs the field number and the wire type together. The formula is tagValue = (fieldNumber << 3) | wireType . The field number occupies the upper 29 bits, and the wire type occupies the lower 3 bits . The tag is then varint-encoded, so small field numbers produce short tags and large field numbers produce longer ones.

The wire type tells the parser how to read the value that follows. Protobuf defines six wire types, two of which are deprecated :

Wire Type	Value	Used By
Varint	0	`int32`, `int64`, `uint32`, `uint64`, `sint32`, `sint64`, `bool`, `enum`
64-bit	1	`fixed64`, `sfixed64`, `double`
Length-delimited	2	`string`, `bytes`, embedded messages, packed repeated fields
Start group	3	Deprecated in proto3
End group	4	Deprecated in proto3
32-bit	5	`fixed32`, `sfixed32`, `float`

A string "hello" in field 1 encodes as 0x0A 0x05 0x68 0x65 0x6C 0x6C 0x6F. The 0x0A is the tag: field number 1 shifted left by 3 is 0x08, plus wire type 2 is 0x0A. The 0x05 is the length, and the remaining bytes are the ASCII values . A varint value of 150 encodes as 0x96 0x01: the first byte carries seven data bits and a continuation flag, and the second byte carries the remaining bits .

b. Field numbers and the cost of encoding

Field numbers 1 through 15 produce a one-byte tag because the tag value fits in seven bits. Numbers 16 through 2047 produce a two-byte tag, and numbers 2048 through 262,143 produce three bytes . The practical rule is to reserve 1–15 for the most frequently populated fields, because those fields pay the tag cost on every message .

The valid field number range is 1 to 536,870,911 (2²⁹ − 1), with 19,000 to 19,999 reserved by the Protobuf implementation and 536,870,912 and above reserved for future use . A field number cannot be zero; zero is not a valid field identifier.

The choice of integer type affects size for negative values. int32 and int64 use varint encoding, which is efficient for small positive numbers but inefficient for negative numbers: a negative int32 is encoded as a 64-bit unsigned value, producing a ten-byte varint . sint32 and sint64 use ZigZag encoding, which maps signed values to unsigned values such that small negatives become small positives, and are the correct choice when negative values are expected .

c. Compatibility rules: safe, risky, and breaking

The compatibility taxonomy determines which schema changes can ship without a coordinated migration. The categories are based on whether old readers can parse new data and new readers can parse old data.

Safe changes are those that neither break the wire format nor change the meaning of existing data. Adding a new field with a new number is safe because old readers skip unknown fields and new readers apply the default when the field is absent . Adding a new enum value is safe at the wire level, though it may break exhaustive switches in generated code . Adding a new message or service is safe.

Wire-safe but code-breaking changes parse correctly but change generated APIs or behavior. Adding optional to an existing scalar field is wire-compatible but changes the generated API from a plain field to a presence-tracking field . Removing a field is wire-safe if the number is reserved, but it breaks code that references the field . Converting between int32 and int64 is wire-compatible in one direction but may truncate in the other .

Breaking changes corrupt data or lose information. Renumbering a field changes its wire identity; an old reader will interpret the bytes under the wrong field number . Changing a field’s type incompatibly—string to int32, for example—means the old reader interprets the bytes with the wrong wire type and decoder . Reusing a deleted field number is the most dangerous: old data with that number may still exist in logs, queues, or databases, and a new field with the same number will be decoded as the old type .

The deletion pattern is therefore a lifecycle, not a single step. Mark the field deprecated, wait until metrics show it is unused, remove it from the schema, and add its number and name to a reserved block . The reserved statement is enforced by protoc at compile time: any future schema that tries to use the reserved number or name fails to compile . The reservation is permanent.

Complete Example Session

// ============================================
// PART 1: FIELD NUMBERS AND TAG COST
// ============================================
syntax = "proto3";

message User {
  string id = 1;          // tag: 1 byte (field 1-15)
  string email = 2;       // tag: 1 byte
  int32 age = 15;         // tag: 1 byte (last of the cheap range)
  string bio = 16;        // tag: 2 bytes (field 16+)
  sint32 delta = 17;      // ZigZag, efficient for negatives
  bytes avatar = 18;      // length-delimited
}

// ============================================
// PART 2: WIRE ENCODING OF ONE FIELD
// ============================================
// User { email: "a@b.com" }
//
// Field 2, wire type 2 (length-delimited)
// tag = (2 << 3) | 2 = 18 = 0x12
// length = 7 = 0x07
// payload = "a@b.com" = 61 40 62 2E 63 6F 6D
//
// bytes: 12 07 61 40 62 2E 63 6F 6D

// ============================================
// PART 3: SAFE ADDITION
// ============================================
message UserV2 {
  string id = 1;
  string email = 2;
  int32 age = 15;
  string bio = 16;
  sint32 delta = 17;
  bytes avatar = 18;

  // New field: new number, no conflict
  string display_name = 19;
}
// Old readers skip field 19; new readers default it when absent.

// ============================================
// PART 4: DELETION AND RESERVATION
// ============================================
message UserV3 {
  reserved 18, "avatar";  // permanent: number and name retired

  string id = 1;
  string email = 2;
  int32 age = 15;
  string bio = 16;
  sint32 delta = 17;
  string display_name = 19;
  // Field 18 cannot be reused; protoc rejects it.
}

The parts covered tag cost by field number, the wire encoding of a single length-delimited field, a safe field addition, and the reservation pattern after deletion.

Quick Reference

Tag Encoding

Component	Bits	Formula
Field number	Upper 29	`tag = (fieldNumber << 3) \| wireType`
Wire type	Lower 3	Values 0, 1, 2, 5 (3/4 deprecated)

Field Number Ranges

Range	Tag Size	Use
1–15	1 byte	Frequent fields
16–2047	2 bytes	Standard fields
2048–262,143	3 bytes	Rare fields
19,000–19,999	—	Reserved by protobuf
536,870,911	Max	Upper bound

Compatibility Taxonomy

Change	Wire	Code	Action
Add field (new number)	Safe	Safe	Ship
Add `optional` to scalar	Safe	Breaking	Stage rollout
Remove field	Safe if reserved	Breaking	Reserve number + name
Renumber field	Breaking	Breaking	Never
Change type incompatibly	Breaking	Breaking	Never
Reuse deleted number	Breaking	Breaking	Never

Best Practices

✅ Do This:

// Reserve 1-15 for frequent fields
string id = 1;                                               // ✅

// Use sint32 for negative values
sint32 delta = 2;                                            // ✅

// Reserve after deletion
reserved 5, "old_field";                                     // ✅

// Add new fields with new numbers
string new_field = 6;                                        // ✅

❌ Don’t Do This:

// Reuse a field number
string replacement = 5;  // field 5 was deleted                // ❌

// Change type incompatibly
int32 id = 1;  // was string                                  // ❌

// Renumber a field
string email = 10;  // was 2                                 // ❌

// Remove without reserving
// (number becomes available for reuse)                       // ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Silent data corruption	Field number reused	Reserve deleted numbers permanently
Negative numbers large on wire	`int32` used	Use `sint32`/`sint64`
Old reader crashes on new data	Type changed incompatibly	Add new field instead
`optional` breaks generated code	Presence changes API	Stage readers before writers
Field number typo wastes bytes	16 used instead of 15	Check the one-byte range

Real-World Examples

1. Tag formula

(fieldNumber << 3) | wireType

2. String encoding

0A 05 68 65 6C 6C 6F  // field 1, "hello"

3. Varint encoding

96 01  // 150

4. Frequent field range

string id = 1;  // 1-byte tag

5. Negative-efficient type

sint32 delta = 2;

6. Reserved number

reserved 5;

7. Reserved name

reserved "old_field";

8. Safe addition

string new_field = 6;

9. Breaking type change

// string id = 1;  →  int32 id = 1;  // NEVER

10. Deprecation marker

int32 old = 5 [deprecated = true];

Visual

The Tag Structure

┌─────────────────────────────────────────────────────────┐
│  TAG = (field_number << 3) | wire_type                  │
│                                                         │
│  Field 2, wire type 2 (length-delimited):               │
│  (2 << 3) | 2 = 16 | 2 = 18 = 0x12                      │
│                                                         │
│  Wire bytes: 0x12 [length] [payload]                    │
│                                                         │
│  Field 1, wire type 0 (varint):                         │
│  (1 << 3) | 0 = 8 = 0x08                                │
│                                                         │
│  Wire bytes: 0x08 [varint value]                        │
└─────────────────────────────────────────────────────────┘

Tag Cost by Field Number

┌─────────────────────────────────────────────────────────┐
│  FIELD 1-15      → 1 byte tag                           │
│  FIELD 16-2047   → 2 byte tag                           │
│  FIELD 2048+     → 3 byte tag                           │
│                                                         │
│  Every message pays the tag cost for every field        │
│  Put frequently-set fields in 1-15                      │
└─────────────────────────────────────────────────────────┘

Compatibility Spectrum

┌─────────────────────────────────────────────────────────┐
│  SAFE                    RISKY              BREAKING    │
│  ────                    ─────              ────────    │
│  Add field               Add optional       Renumber    │
│  Add enum value          Remove (reserve)   Change type │
│  Add message             int32 ↔ int64      Reuse number│
│                                                         │
│  Wire-safe and code-safe  Wire-safe only    Neither     │
└─────────────────────────────────────────────────────────┘

Deletion Lifecycle

┌─────────────────────────────────────────────────────────┐
│  1. Mark deprecated                                     │
│     int32 old = 5 [deprecated = true];                  │
│                                                         │
│  2. Wait for zero usage                                 │
│     (metrics, logs, queue inspection)                   │
│                                                         │
│  3. Remove and reserve                                  │
│     reserved 5, "old";                                  │
│                                                         │
│  4. Number is retired forever                           │
│     protoc rejects any reuse                            │
└─────────────────────────────────────────────────────────┘

Summary

Item	Value
Tag formula	`(fieldNumber << 3) \| wireType`
Wire types	0 varint, 1 64-bit, 2 length-delimited, 5 32-bit
One-byte tag range	Fields 1–15
Max field number	536,870,911
Reserved range	19,000–19,999
Negative-efficient type	`sint32`/`sint64` (ZigZag)
Safe addition	New field, new number
Breaking change	Renumber, type change, reuse
Deletion	Reserve number and name permanently

Key takeaways:

The wire format is numbers, not names. The tag packs the field number and wire type into a varint; the field name exists only in the schema and generated code. This is why renaming is wire-safe and renumbering is catastrophic .
Field numbers 1–15 are cheaper. The tag fits in one byte for those numbers, and every message pays that cost. Reserve the low range for the most frequently populated fields .
int32 is inefficient for negatives. A negative int32 becomes a ten-byte varint; sint32 uses ZigZag encoding to keep small negatives small .
Compatibility has three tiers. Adding a field is safe; adding optional is wire-safe but code-breaking; renumbering and type changes are breaking. The tier determines the rollout strategy .
Deleted field numbers are retired forever. The reserved statement enforces this at compile time, preventing the silent corruption that occurs when a new field inherits an old number that still exists in persisted data .

Remember: Protobuf’s compatibility guarantee is a consequence of its wire format, not a separate feature. The field number is the stable identity; the field name is a convenience. When changing a schema, the question is not “does this compile?” but “what does an old reader do with this data?” Adding a field with a new number is invisible to old readers. Renumbering a field makes old readers decode the bytes as the wrong type. Reserve deleted numbers, use sint for negatives, and put frequent fields in the one-byte range. The wire format is the contract.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!