JSON 04 📄 String Escaping and Character Encoding (UTF-8) - KandZ

⏱️ Estimated read: 13 min | |

A JSON string is a sequence of Unicode characters, and the JSON text that carries it must be encoded in UTF-8 when it is exchanged between systems that are not part of a closed ecosystem. The RFC 8259 makes this requirement explicit: “JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8” . The encoding is the layer that turns the string into bytes for transport, and the escaping is the layer that lets a string contain characters that cannot appear literally. The two are separate concerns, and the interaction between them is where the interoperability problems live.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

The escaping rules are small but precise. The quote and the backslash must be escaped. The control characters from U+0000 to U+001F must be escaped. The solidus may be escaped. The \uXXXX form is the escape for any character in the Basic Multilingual Plane, and the characters outside the BMP—the emoji, the musical symbols, the rare CJK ideographs—are represented as a UTF-16 surrogate pair, a twelve-character sequence of two \uXXXX escapes . The encoding is UTF-8, and the escape is the ASCII-safe representation. The two are not the same thing, and the distinction matters for the parsers and the databases.

Key point: JSON strings are Unicode, and the JSON text must be UTF-8 for the interoperable exchange. The escaping is the ASCII-safe representation, and the characters outside the BMP use the UTF-16 surrogate pair. The unpaired surrogates are allowed by the ABNF but unpredictable in the behavior, and the implementations must validate them or replace them.

Why the escaping and the encoding are separate

The transport problem. The JSON text is a sequence of bytes, and the bytes must be interpreted as characters. The UTF-8 is the encoding that the RFC requires, and it is the encoding that every parser accepts. The escaping is the mechanism that represents the characters that the JSON grammar cannot carry literally—the quote, the backslash, the control characters—and the characters that the transport cannot carry safely in some contexts. The escaped string is ASCII, and the ASCII is the subset of UTF-8. The escaping is the transport-safe representation, and the encoding is the interpretation of the bytes.

The surrogate pair problem. The characters outside the BMP—the code points from U+10000 to U+10FFFF—are not representable in a single UTF-16 code unit. The UTF-16 represents them as a pair of code units: a high surrogate from U+D800 to U+DBFF and a low surrogate from U+DC00 to U+DFFF. The JSON escape represents the pair as two \uXXXX sequences . The pair is the representation, and the character is the abstract concept. The parser must combine the pair into the single code point, and the serializer must split the code point into the pair. The Apache Thrift issue THRIFT-6122 is the concrete example of the failure: the Ruby JSON protocol converts the high surrogate independently, and Ruby rejects the standalone surrogate with a RangeError before the low surrogate can be combined .

The unpaired surrogate problem. The ABNF grammar allows a lone \uDEAD—a single unpaired surrogate—because the grammar does not enforce the pair structure. The RFC 8259 warns that the behavior is unpredictable: the implementations “might return different values for the length of a string value, or even suffer fatal runtime exceptions” . The unpaired surrogate is not a Unicode character, and it has no UTF-8 encoding. The implementation that receives it may reject it, replace it with the U+FFFD replacement character, or store the raw code unit. The Apache Flink issue FLINK-41204 is the concrete example: the VARIANT parser replaces the unpaired surrogate with the ? character, which means the stored text is different from the input and nothing fails . The three behaviors are all valid, and the incompatibility is the problem.

The trade-off. The escaping is the mechanism that makes the string transport-safe, but it is not free. The escape sequences are longer than the literal characters, and the parsing is slower because the escapes must be decoded. The UTF-8 encoding is the efficient transport, and the escaping is the safety net for the characters that the grammar or the transport cannot carry. The two are the layers, and the design is the choice of which layer handles which character.

a. The escape sequences and the Unicode representation

The escape sequences are the backslash-prefixed forms. The \" is the double quote, the \\ is the reverse solidus, the \/ is the solidus, the \b is the backspace, the \f is the form feed, the \n is the line feed, the \r is the carriage return, and the \t is the tab. The \uXXXX is the escape for the four-hex-digit code unit . The escapes are the grammar’s mechanism for the characters that must be escaped, and the \uXXXX is the general form.

// The escape sequences
const escaped = "He said \"hello\" and \\ backslash";
// The result: He said "hello" and \ backslash

// The control character escape
const control = "Line 1\nLine 2";
// The result: the newline between the lines

// The Unicode escape
const unicode = "\u0041"; // The 'A'

The characters outside the BMP are escaped as the surrogate pair. The G clef (U+1D11E) is \uD834\uDD1E, and the grinning face (U+1F600) is \uD83D\uDE00 . The pair is the UTF-16 representation, and the JSON escape is the transport-safe form. The parser combines the pair into the single code point, and the string comparison is the code point comparison, not the code unit comparison .

// The surrogate pair
const clef = "\uD834\uDD1E"; // The G clef (U+1D11E)
const grin = "\uD83D\uDE00"; // The grinning face (U+1F600)

// The parser combines the pair into the single character.
console.log(clef.length); // 2 (the UTF-16 code units)
console.log([...clef].length); // 1 (the code point)

The escape is not the only representation. The JSON text can contain the literal UTF-8 characters, and the parser accepts them if the text is valid UTF-8. The literal emoji and the escaped pair are the same character, and the parser produces the same code point. The escaping is the choice of the transport, not the choice of the data model.

b. The UTF-8 encoding and the byte order

UTF-8 is a variable-width encoding. The ASCII characters (U+0000 to U+007F) are a single byte, and the bytes are identical to the ASCII encoding. The code points from U+0080 to U+07FF are two bytes, the code points from U+0800 to U+FFFF are three bytes, and the code points above U+FFFF are four bytes. The first byte indicates the number of the bytes that follow, which makes the forward parsing efficient and the self-synchronization possible.

The UTF-8 has no byte order problem. The code unit is a single byte, and the byte order is not applicable. The UTF-16 and the UTF-32 have the byte order problem, and the byte order mark (BOM) is the mechanism that distinguishes the big-endian and the little-endian. The RFC 8259 states that the implementations must not add a BOM to the beginning of the network-transmitted JSON text, and the parsers may ignore the BOM if it is present . The distinction between the network-transmitted text and the file content is the nuance: the RFC’s requirement applies to the text exchanged between the systems, not necessarily to the local files. The C++ SG16 discussion notes that the RFC permits the non-UTF-8 encodings in a closed system and permits the BOM in the file contents, while the network transmission requires the UTF-8 without the BOM .

// The UTF-8 encoding of the 'A'
const a = "A";
const bytes = new TextEncoder().encode(a);
// Uint8Array [ 65 ] (the single byte)

// The UTF-8 encoding of the 'é' (U+00E9)
const e = "é";
const eBytes = new TextEncoder().encode(e);
// Uint8Array [ 195, 169 ] (the two bytes)

// The UTF-8 encoding of the '😀' (U+1F600)
const emoji = "😀";
const emojiBytes = new TextEncoder().encode(emoji);
// Uint8Array [ 240, 159, 152, 128 ] (the four bytes)

The encoding is the layer beneath the string. The JSON parser decodes the bytes as UTF-8, and the string it produces is a sequence of the code points. The escaping is the layer above the string, and the escape sequence is decoded into the code point. The two layers are the transport and the representation, and the parser is the mechanism that connects them.

Complete Example Session

// ============================================
// PART 1: THE ESCAPE SEQUENCES
// ============================================

const escaped = "The quote is \" and the backslash is \\";
// The \" is the double quote.
// The \\ is the reverse solidus.

// ============================================
// PART 2: THE CONTROL CHARACTERS
// ============================================

const control = "Line 1\nLine 2\tTabbed";
// The \n is the line feed.
// The \t is the tab.
// The unescaped control characters are not allowed.

// ============================================
// PART 3: THE UNICODE ESCAPE
// ============================================

const a = "\u0041"; // The 'A'
const e = "\u00E9"; // The 'é'

// The \uXXXX is the four-hex-digit code unit.

// ============================================
// PART 4: THE SURROGATE PAIR
// ============================================

const clef = "\uD834\uDD1E"; // The G clef (U+1D11E)
const grin = "\uD83D\uDE00"; // The grinning face (U+1F600)

// The pair is the UTF-16 representation.
// The parser combines the pair into the single code point.

// ============================================
// PART 5: THE UNPAIRED SURROGATE
// ============================================

const lone = "\uDEAD"; // The unpaired low surrogate

// The ABNF allows it.
// The behavior is unpredictable.
// The parser may reject it, replace it, or store it.

// ============================================
// PART 6: THE UTF-8 ENCODING
// ============================================

const encoder = new TextEncoder();
const bytes = encoder.encode("A"); // Uint8Array [ 65 ]
const emojiBytes = encoder.encode("😀"); // Uint8Array [ 240, 159, 152, 128 ]

// The UTF-8 is the variable-width encoding.
// The ASCII is the single byte.
// The emoji is the four bytes.

// ============================================
// PART 7: THE BOM
// ============================================

// The RFC 8259 requires no BOM for the network-transmitted text.
// The implementations may ignore the BOM if present.
// The BOM is the U+FEFF.

// ============================================
// PART 8: THE STRING COMPARISON
// ============================================

// The escaped and the literal are the same string.
const escapedString = "a\\b"; // The backslash
const unicodeString = "a\u005Cb"; // The same backslash
console.log(escapedString === unicodeString); // true (after the parse)

// The comparison is the code point comparison.
// The unescaped comparison is the bug.

// ============================================
// PART 9: THE VALIDATION
// ============================================

// The unpaired surrogate should be rejected.
const invalid = '{"name": "\\uDEAD"}';
try {
  const parsed = JSON.parse(invalid);
  // The behavior is implementation-dependent.
} catch (e) {
  // Some parsers throw.
}

// ============================================
// PART 10: THE CHECKLIST
// ============================================

// [ ] The JSON text is UTF-8 encoded.
// [ ] The control characters are escaped.
// [ ] The quote and the backslash are escaped.
// [ ] The characters outside the BMP use the surrogate pair.
// [ ] The unpaired surrogates are rejected or replaced.
// [ ] The string comparison is the code point comparison.
// [ ] The BOM is not added to the network-transmitted text.

The ten parts covered the escape sequences, the control characters, the Unicode escape, the surrogate pair, the unpaired surrogate, the UTF-8 encoding, the BOM, the string comparison, the validation, and the checklist. The through-line is that the escaping is the transport-safe representation, and the encoding is the interpretation of the bytes.

Quick Reference

The escape sequences

Escape	Character	Code point
`\"`	The double quote	U+0022
`\\`	The reverse solidus	U+005C
`\/`	The solidus	U+002F
`\b`	The backspace	U+0008
`\f`	The form feed	U+000C
`\n`	The line feed	U+000A
`\r`	The carriage return	U+000D
`\t`	The tab	U+0009
`\uXXXX`	The code unit	U+XXXX

The UTF-8 byte ranges

Code point range	Bytes	Example
U+0000–U+007F	1	`A` → `65`
U+0080–U+07FF	2	`é` → `195, 169`
U+0800–U+FFFF	3	`€` → `226, 130, 172`
U+10000–U+10FFFF	4	`😀` → `240, 159, 152, 128`

The surrogate pair

Code point	Surrogate pair
U+1D11E (G clef)	`\uD834\uDD1E`
U+1F600 (grinning face)	`\uD83D\uDE00`

Best Practices

✅ Do This:

// Encode the JSON text as UTF-8
const json = JSON.stringify(data);
const bytes = new TextEncoder().encode(json);                     // ✅

// Escape the control characters
const escaped = "Line 1\nLine 2";                                // ✅

// Use the surrogate pair for the characters outside the BMP
const emoji = "\uD83D\uDE00";                                    // ✅

❌ Don’t Do This:

// Add a BOM to the network-transmitted JSON text
const bom = "\uFEFF" + JSON.stringify(data);                     // ❌

// Use the unpaired surrogate
const lone = "\uDEAD"; // the unpredictable behavior              // ❌

// Compare the escaped and the unescaped strings without parsing
"a\\b" === "a\u005Cb"; // false before the parse                 // ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
The string length is wrong	The unpaired surrogate	Reject or replace the surrogate
The comparison fails	The escaped and the unescaped forms	Parse both before the comparison
The emoji is corrupted	The wrong encoding or the split surrogate	Use the UTF-8 and combine the pair
The BOM breaks the parser	The BOM is added	Remove the BOM for the network transmission

Real-World Examples

1. The basic escape

const s = "He said \"hello\"";

2. The control character

const s = "Line 1\nLine 2";

3. The Unicode escape

const s = "\u0041"; // 'A'

4. The surrogate pair

const s = "\uD834\uDD1E"; // The G clef

5. The unpaired surrogate

const s = "\uDEAD"; // The unpredictable behavior

6. The UTF-8 encoding

const bytes = new TextEncoder().encode("😀");

7. The BOM

const bom = "\uFEFF"; // Not for the network transmission

8. The string comparison

JSON.parse('"a\\\\b"') === JSON.parse('"a\\u005Cb"'); // true

9. The validation

try { JSON.parse('{"name":"\\uDEAD"}'); } catch (e) { /* rejected */ }

10. The Flink case

// The unpaired surrogate is replaced with '?'

Visual

The Escape Sequences

┌──────────────────────────────────────────────────────────────┐
│  THE ESCAPE SEQUENCES                                        │
│                                                              │
│  \"  →  the double quote                                    │
│  \\  →  the reverse solidus                                 │
│  \n  →  the line feed                                       │
│  \t  →  the tab                                             │
│  \u0041  →  the 'A' (the Unicode code point)                │
│                                                              │
│  The unescaped control characters are not allowed.          │
└──────────────────────────────────────────────────────────────┘

The Surrogate Pair

┌──────────────────────────────────────────────────────────────┐
│  THE CHARACTERS OUTSIDE THE BMP                              │
│                                                              │
│  The G clef (U+1D11E)                                       │
│  ┌─────────────────────┐                                    │
│  │ \uD834  \uDD1E      │  The surrogate pair                │
│  │ (high)  (low)       │                                    │
│  └──────────┬──────────┘                                    │
│             │                                                │
│             ▼                                                    │
│  ┌─────────────────────┐                                    │
│  │ U+1D11E             │  The single code point             │
│  └─────────────────────┘                                    │
│                                                              │
│  The parser combines the pair.                              │
│  The serializer splits the code point.                      │
└──────────────────────────────────────────────────────────────┘

The UTF-8 Byte Ranges

┌──────────────────────────────────────────────────────────────┐
│  THE VARIABLE-WIDTH ENCODING                                 │
│                                                              │
│  'A'  (U+0041)  →  1 byte   →  65                           │
│  'é'  (U+00E9)  →  2 bytes  →  195, 169                     │
│  '€'  (U+20AC)  →  3 bytes  →  226, 130, 172                │
│  '😀' (U+1F600) →  4 bytes  →  240, 159, 152, 128           │
│                                                              │
│  The first byte indicates the number of bytes that follow.  │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
JSON encoding	UTF-8 for the interoperable exchange
Escape sequences	`\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`, `\uXXXX`
Surrogate pair	The characters outside the BMP use the two `\uXXXX` escapes
Unpaired surrogate	Allowed by the ABNF, unpredictable in the behavior
UTF-8 byte ranges	1 byte (ASCII), 2, 3, 4 bytes
BOM	Not added to the network-transmitted text, may be ignored by the parser

Key takeaways:

The JSON text must be UTF-8 for the interoperable exchange. The RFC 8259 requires the UTF-8, and the previous specifications allowed the UTF-16 and the UTF-32 . The UTF-8 is the only encoding that every parser accepts.
The escape sequences are the transport-safe representation. The \", \\, and the control character escapes are the grammar’s mechanism for the characters that cannot appear literally. The \uXXXX is the general form .
The characters outside the BMP use the surrogate pair. The G clef is \uD834\uDD1E, and the emoji is the pair of the high and the low surrogates . The parser combines the pair into the single code point.
The unpaired surrogate is the interoperability hazard. The ABNF allows it, but the behavior is unpredictable: the implementations may reject it, replace it with the U+FFFD, or store the raw code unit . The Flink issue shows the replacement with ? , and the Thrift issue shows the RangeError .
The string comparison is the code point comparison. The escaped and the unescaped forms are the same string after the parse, and the comparison must be on the parsed values, not the raw text .

Remember: JSON strings are Unicode, and the JSON text is UTF-8. The escaping is the transport-safe representation, and the surrogate pair is the representation of the characters outside the BMP. The unpaired surrogate is the grammar’s edge and the implementation’s hazard. The encoding is the interpretation of the bytes, and the escaping is the interpretation of the characters. The two are the layers, and the parser is the connection.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!