Serialization 15 📄 Canonical Serialization — Deterministic Output for Hashing and Signing - KandZ

⏱️ Estimated read: 10 min | |

A hash or a signature is only as trustworthy as the bytes it was computed over. If the same logical data can serialize to different byte sequences—different key order, different whitespace, different number formatting—then a signature computed over one sequence will fail verification against another, even though the data is identical. Canonical serialization is the discipline of defining exactly one byte sequence for a given logical value, so that independent implementations, on different platforms, at different times, produce the same output and therefore the same hash or signature.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

This chapter covers the canonicalization schemes for the major formats, the properties that make a serialization canonical, and the limitations that make “deterministic” different from “canonical.” The sections that follow walk through the JSON Canonicalization Scheme, the CBOR deterministic encoding profiles, and the cases where the serializer’s own determinism is not enough.

Key point: Canonical serialization defines one byte sequence per logical value; a serializer that is deterministic within one process is not necessarily canonical across processes, languages, or schema versions, and cryptographic systems must use a canonicalization specification rather than a serializer’s “deterministic” flag.

a. JSON Canonicalization Scheme (RFC 8785)

The JSON Canonicalization Scheme (JCS) is the IETF specification for producing a canonical JSON byte sequence. It builds on ECMAScript’s number serialization and I-JSON restrictions to eliminate the variability that JSON’s grammar permits .

The input constraints are strict. The data must be I-JSON compatible: no duplicate property names, all strings expressible as Unicode, and all numbers expressible as IEEE 754 double-precision values . Applications that need higher precision or longer integers must represent them as JSON strings, because JCS serializes numbers using the ECMAScript JSON.stringify algorithm, which is bounded by double precision .

The canonicalization steps are mechanical. Whitespace between tokens is not emitted. Object properties are sorted lexicographically by their UTF-16 code units. Numbers are serialized in the shortest form that round-trips, using the Ryu algorithm. Strings are escaped according to ECMAScript’s rules, with control characters and quotes escaped and everything else emitted as UTF-8 . The output is valid JSON, but it is a single line with sorted keys and normalized numbers—readable by a parser, not by a human.

// Input (key order arbitrary, whitespace present)
{"b":2, "a":1, "c":{"y":3, "x":2}}

// Canonical output (sorted keys, no whitespace)
{"a":1,"b":2,"c":{"x":2,"y":3}}

JCS is not on the IETF standards track, but conformant implementations are expected to adhere to its behavior for security and interoperability . The specification includes sanity checks that must be performed before signing: parse the JSON, verify it adheres to I-JSON, verify the data against the ecosystem’s conventions, and abort if any step fails . The canonical bytes are the signing preimage; the signature is computed over those bytes, not over the original input.

b. CBOR deterministic encoding (CDE)

CBOR’s canonicalization story is more complicated than JSON’s because CBOR was designed to allow encoding variation for constrained environments. The same logical value can be encoded with shortest or longest integers, definite or indefinite lengths, and half, single, or double precision floats. RFC 8949 defines “Preferred Serialization” rules that eliminate most of this variation, and the CBOR Common Deterministic Encoding Profile (CDE) formalizes them for interoperability .

CDE requires Preferred Serialization (shortest argument form, preferred float representation) and adds the constraint of not using indefinite-length encoding . The CDE specification clarifies that NaNs are encoded with the preferred representation, typically 0xf97e00 for the quiet, non-negative NaN with zero payload . The result is a byte sequence that is stable across implementations for the same data model value.

The limitation is that CBOR’s determinism is at the serialization layer, not the data model layer. The CDE draft distinguishes between the information model, the data model, and the serialization. Two values that are equivalent at the data model level—a float 1.0 and an integer 1—are not necessarily equivalent at the serialization layer, and CDE does not attempt to unify them . A protocol that needs the unification must define it at the application layer (the “ALDR” rules in the CBOR drafts) and restrict the set of values its applications use.

c. The limits of “deterministic” serializers

The most dangerous misconception in canonical serialization is that a serializer’s “deterministic” option produces canonical output. It does not.

The Go Protobuf documentation is explicit. MarshalOptions{Deterministic: true} guarantees that the same message serialized repeatedly within the same binary returns the same bytes, and that different processes of the same binary on different machines serialize equal messages to the same bytes . But the documentation continues: “the deterministic serialization is NOT canonical across languages. It is not guaranteed to remain stable over time. It is unstable across different builds with schema changes due to unknown fields. Users who need canonical serialization (e.g., persistent storage in a canonical form, fingerprinting, etc.) must define their own canonicalization specification and implement their own serializer rather than relying on this API” .

The reason is that Protobuf’s wire format allows map entries to be serialized in any order, and unknown fields to be retained or discarded depending on the schema and the runtime. The deterministic flag sorts map entries by key, but that is an implementation detail subject to change, not a specification . A signature computed over Protobuf bytes is not portable across languages or schema versions unless the application defines a canonical form that fixes the map ordering, the unknown field handling, and the field number ordering.

The pattern is general. A canonicalization scheme must be specified externally to the serializer, implemented against the specification rather than the serializer’s options, and tested against reference vectors that an independent implementation can reproduce. The ETERNITY Layer’s EL-C14N-v2 appendix defines exactly this: a normative algorithm (sort keys, minimal numbers, no whitespace, domain tag prefix), determinism tests (re-encode twice, compare bytes; vary whitespace and key order, compare canonical digests), and official reference vectors for compliance . The signature is computed over the canonical bytes, and a verifier that produces different bytes from the same logical input fails the check.

Complete Example Session

# ============================================
# PART 1: THE SIGNATURE PROBLEM
# ============================================
# Signer serializes: {"b":2,"a":1}
# Hash: SHA256('{"b":2,"a":1}')
#
# Verifier parses, re-serializes: {"a":1,"b":2}
# Hash: SHA256('{"a":1,"b":2}')
#
# The hashes differ. The signature fails.
# The data is identical. The bytes are not.

# ============================================
# PART 2: JCS CANONICALIZATION
# ============================================
# Both sides canonicalize before signing/verifying:
# Input:  {"b":2, "a":1}
# Canonical: {"a":1,"b":2}
#
# Hash: SHA256('{"a":1,"b":2}')
# The hashes match. The signature verifies.

# ============================================
# PART 3: JCS RULES
# ============================================
# 1. No whitespace
# 2. Object keys sorted lexicographically
# 3. Numbers in shortest round-trip form (Ryu)
# 4. Strings escaped per ECMAScript rules
# 5. I-JSON constraints (no duplicate keys, Unicode, double precision)

# ============================================
# PART 4: PROTOBUF'S DETERMINISTIC FLAG IS NOT CANONICAL
# ============================================
# MarshalOptions{Deterministic: true}
#   → same bytes within the same binary
#   → NOT canonical across languages
#   → NOT stable across schema changes
#   → NOT suitable for signatures without a separate spec

# ============================================
# PART 5: THE CANONICALIZATION CONTRACT
# ============================================
# A canonicalization scheme must define:
# - The input constraints (I-JSON, no duplicate keys)
# - The normalization rules (sort order, number format)
# - The output encoding (UTF-8, no BOM)
# - The domain separation tag (prevent cross-protocol reuse)
# - The reference vectors (for independent verification)

The parts covered the signature problem, the JCS fix, the JCS rules, the Protobuf limitation, and the canonicalization contract.

Quick Reference

Canonicalization Schemes

Format	Scheme	Status
JSON	JCS (RFC 8785)	Informational
CBOR	CDE (draft)	Draft
Protobuf	No standard	Must define your own
XML	Canonical XML (C14N)	W3C Recommendation

JCS Rules

Rule	Detail
Whitespace	None emitted
Key order	Lexicographic by UTF-16 code units
Numbers	ECMAScript `JSON.stringify` (Ryu)
Strings	ECMAScript escape rules
Input	I-JSON (no duplicate keys, double precision)

The “Deterministic” Trap

Claim	Reality
Protobuf `Deterministic: true`	Same bytes within one binary
Cross-language canonical	No
Stable across schema changes	No
Suitable for signatures	Only with a separate spec

Best Practices

✅ Do This:

# Use JCS for JSON signing
# Canonicalize before hashing/signing                            # ✅

# Define the canonicalization spec explicitly
# Sort keys, normalize numbers, no whitespace                   # ✅

# Test with reference vectors
# Independent implementation must produce same bytes            # ✅

# Use domain separation tags
# Prepend a fixed tag to prevent cross-protocol reuse           # ✅

❌ Don’t Do This:

# Sign the original serialization
# (key order and whitespace vary)                               # ❌

# Trust Protobuf's Deterministic flag for signatures
# (not canonical across languages or schema versions)           # ❌ 

# Assume deterministic = canonical
# (deterministic within one binary ≠ canonical across systems)  # ❌

# Skip the input constraints
# (duplicate keys, non-double numbers break JCS)                # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Signature fails after re-serialization	Non-canonical input	Canonicalize before signing
Cross-language verification fails	Serializer-specific determinism	Use a canonicalization spec
Hash changes after schema update	Unknown fields retained	Fix the canonical form
Duplicate keys pass silently	Parser keeps last value	Reject duplicates (JCS)

Real-World Examples

1. JCS canonical JSON

{"a":1,"b":2,"c":{"x":2,"y":3}}

2. Non-canonical input

{"b":2, "a":1}

3. JCS signing preimage

SHA256(canonical_bytes)

4. Protobuf deterministic flag

MarshalOptions{Deterministic: true}

5. Protobuf limitation

NOT canonical across languages

6. Domain tag prefix

EL-C14N-v2||{...}

7. CBOR preferred serialization

Shortest integer form, definite length

8. Canonical XML

W3C C14N, namespace-aware

9. Reference vectors

Input → expected canonical bytes → assert equality

10. Input constraint

Reject duplicate keys before canonicalization

Visual

The Signature Problem

┌─────────────────────────────────────────────────────────┐
│  SIGNER                                                 │
│  {"b":2,"a":1} → SHA256 → signature                     │
│                                                         │
│  VERIFIER                                               │
│  parse → {"a":1,"b":2} → SHA256 → mismatch              │
│                                                         │
│  Same data, different bytes, failed verification        │
└─────────────────────────────────────────────────────────┘

The Canonicalization Fix

┌─────────────────────────────────────────────────────────┐
│  SIGNER                                                 │
│  input → canonicalize → {"a":1,"b":2} → SHA256 → sig    │
│                                                         │
│  VERIFIER                                               │
│  input → canonicalize → {"a":1,"b":2} → SHA256 → match  │
│                                                         │
│  Canonical bytes are the signing preimage               │
└─────────────────────────────────────────────────────────┘

Deterministic vs Canonical

┌─────────────────────────────────────────────────────────┐
│  DETERMINISTIC                  CANONICAL               │
│                                                         │
│  Same bytes within             Same bytes across        │
│  one binary                    languages, platforms,    │
│                                schema versions           │
│                                                         │
│  Implementation detail         Specification            │
│  May change                    Stable                   │
│  Not for signatures            For signatures           │
└─────────────────────────────────────────────────────────┘

JCS Canonicalization Steps

┌─────────────────────────────────────────────────────────┐
│  1. Parse (I-JSON: no duplicates, double numbers)       │
│         │                                               │
│         ▼                                               │
│  2. Sort object keys lexicographically                  │
│         │                                               │
│         ▼                                               │
│  3. Serialize numbers in shortest form                  │
│         │                                               │
│         ▼                                               │
│  4. Escape strings per ECMAScript                       │
│         │                                               │
│         ▼                                               │
│  5. Emit UTF-8, no whitespace                           │
│         │                                               │
│         ▼                                               │
│  6. Hash/sign the canonical bytes                       │
└─────────────────────────────────────────────────────────┘

Summary

Item	Value
Purpose	One byte sequence per logical value
JSON scheme	JCS (RFC 8785)
CBOR scheme	CDE (Preferred Serialization + no indefinite)
Protobuf scheme	None; define your own
JCS input	I-JSON (no duplicate keys, double precision)
JCS key order	Lexicographic
JCS numbers	ECMAScript stringify (Ryu)
Deterministic ≠ canonical	Protobuf `Deterministic` is per-binary

Key takeaways:

Canonicalization defines one byte sequence per logical value. Without it, key order, whitespace, and number formatting produce different hashes for identical data, and signatures fail verification across implementations .
JCS is the JSON standard. It mandates no whitespace, lexicographic key sorting, ECMAScript number serialization, and I-JSON input constraints. The canonical bytes are the signing preimage, not the original input .
CBOR has Preferred Serialization and CDE. The schemes eliminate most encoding variability, but they operate at the serialization layer, not the data model layer. A protocol that needs data-model equivalence must define it at the application layer .
A serializer’s “deterministic” flag is not canonical. Protobuf’s Deterministic: true guarantees same bytes within one binary, not across languages, platforms, or schema versions. Signatures require a separate canonicalization specification .
Test with reference vectors. A canonicalization scheme is only trustworthy if an independent implementation produces the same bytes for the same input. The ETERNITY Layer’s determinism tests—re-encode twice, vary whitespace and key order, compare canonical digests—are the pattern .

Remember: A signature is a commitment to a byte sequence, not to a data structure. If the byte sequence can vary for the same data, the signature is ambiguous. Canonical serialization removes the ambiguity by specifying the one sequence that every implementation must produce. Use JCS for JSON, CDE for CBOR, and your own specification for Protobuf. Test against reference vectors, reject non-canonical input, and never trust a serializer’s “deterministic” option as a substitute for a canonicalization spec.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!