Serialization 16 📄 Streaming Serialization — JSON Lines, NDJSON, and Incremental Parsing
A JSON array cannot be appended to. The closing bracket ends the document, and adding another object requires rewriting the file or buffering the entire array in memory before writing it out. This is the problem streaming serialization solves. JSON Lines (also called NDJSON or newline-delimited JSON) removes the brackets and uses a newline as the separator between values, so each record is an independent JSON document that can be written, read, and processed without touching the others .
This chapter covers the line-delimited format family, the streaming parsers that consume it incrementally, and the failure modes that appear when a stream is cut mid-record. The sections that follow walk through the format’s structure, the incremental parsing APIs that handle partial input, and the error-recovery patterns that keep a pipeline running when one line is malformed.
Key point: JSON Lines trades the single-document guarantee of JSON for appendability and constant-memory processing; each line is a complete JSON value, and the parser must handle partial lines at chunk boundaries without losing the records that were already complete.
a. The format family: JSON Lines, NDJSON, and JSON Text Sequences
JSON Lines (JSONL) and Newline-Delimited JSON (NDJSON) are effectively the same format with different histories. Each line is one JSON value, separated by \n or \r\n, with no enclosing brackets and no commas between records . The format is not valid JSON as a whole, which is the point: a writer can append a record at any time, and a reader can process records one at a time without loading the file .
The distinction between JSONL and NDJSON is mostly naming. JSON Lines is the older name, NDJSON was forked from it, and the current ecosystem uses both interchangeably . The NDJSON specification removed the comment support that early versions allowed, so a // at the start of a line is not a comment in the current format . The format requires that each line contain no unescaped newlines inside the JSON value; a string with a literal newline must encode it as \n .
JSON Text Sequences (RFC 7464) is a different approach to the same problem. Instead of a newline, it prefixes each JSON text with an ASCII Record Separator character (0x1E) and suggests a trailing line feed . The record separator allows JSON values to contain newlines, which NDJSON forbids, at the cost of a control character that is awkward to edit in a plain text editor . The qlog working group moved from NDJSON to JSON Text Sequences for this reason: it is defined in an RFC and supports newlines within records . For most application logs and data pipelines, JSONL’s simplicity wins; for protocols that need a formal specification and embedded newlines, RFC 7464 is the correct choice.
b. Incremental parsing and the partial-line problem
A streaming parser cannot assume that a chunk from the network or a file read ends on a line boundary. The last line in a chunk may be incomplete, and the parser must report how many bytes it consumed so the caller can keep the remainder for the next chunk.
Bun’s Bun.JSONL.parseChunk is the clearest implementation of this pattern. It returns four properties: values (the records that were fully parsed), read (the number of bytes or characters consumed), done (whether the entire input was consumed), and error (a parse error, or null) . The caller slices off the consumed portion and carries the remainder forward.
// Streaming with Bun.JSONL.parseChunk
let buffer = "";
for await (const chunk of stream) {
buffer += chunk;
const result = Bun.JSONL.parseChunk(buffer);
for (const value of result.values) {
handleRecord(value); // process each complete record
}
// Keep only the unconsumed portion
buffer = buffer.slice(result.read);
}
// Handle any remaining data after the stream ends
if (buffer.length > 0) {
const final = Bun.JSONL.parseChunk(buffer);
for (const value of final.values) handleRecord(value);
if (final.error) console.error("Parse error:", final.error.message);
}The read offset is what makes this work. A parser that returns only the parsed values without telling the caller where it stopped forces the caller to re-parse the entire buffer on each chunk, which is O(n²) in the number of chunks . The offset allows the caller to discard consumed bytes and pass only the remainder to the next parse.
The Go jsonl package applies the same rules at the decoder level. It skips blank lines, trims \r from \r\n endings, and handles a final record without a trailing newline. A single line exceeding the default 16MB limit returns bufio.ErrTooLong, and invalid JSON returns a *DecodeError with the line number . The parser does not throw on the first bad line in a way that discards prior records; the error is per-line, and the caller decides whether to skip or abort.
c. Error recovery and the cost of partial records
The failure modes in a JSONL pipeline are not the same as in a single JSON document. A single JSON document that fails to parse is entirely invalid; a JSONL stream that fails at line 500 has 499 valid records before it. The parser’s error handling determines whether those records reach the consumer.
Bun’s parseChunk does not throw on invalid JSON. It returns the values it parsed before the error and sets the error property. parse, the non-streaming variant, throws only if no values were successfully parsed; if at least one value was parsed before the error, it returns the parsed values without throwing . The asymmetry is deliberate: a stream that produced some data before failing is more useful than an exception that discards it.
The ndjson stream utility in the JSR registry takes a stricter approach by default: a parse failure errors the stream with an NdjsonParseError carrying the 1-indexed line number and the raw line content. The documentation notes that this drops chunks that were queued on the readable side, so the consumer keeps the lines it has already pulled but loses the ones parsed after the most recent pull. For pipelines that need every valid record before a failure, the recommendation is to use a mapSettled recipe instead of letting the stream error .
The practical rule is to decide the error policy before writing the parser. A log-ingestion pipeline that can tolerate dropped malformed lines uses a per-line try/catch and continues. A financial ledger that cannot tolerate gaps stops at the first invalid line and reports the line number. The parser should make both policies possible; the caller chooses.
Complete Example Session
# ============================================
# PART 1: WRITING JSONL INCREMENTALLY
# ============================================
import json
with open("events.jsonl", "a", encoding="utf-8") as f:
for event in generate_events():
# Each write is a complete record followed by a newline
f.write(json.dumps(event) + "\n")
# The file grows line by line; no closing bracket required
# ============================================
# PART 2: READING WITHOUT LOADING THE WHOLE FILE
# ============================================
def process_jsonl(path: str) -> int:
count = 0
with open(path, "r", encoding="utf-8") as f:
for line_num, line in enumerate(f, 1):
line = line.strip()
if not line:
continue # skip blank lines
try:
record = json.loads(line)
handle(record) # process one record
count += 1
except json.JSONDecodeError as e:
print(f"Line {line_num}: {e}") # per-line error
return count
# Memory usage is constant regardless of file size
# ============================================
# PART 3: THE PARTIAL-LINE GUARANTEE
# ============================================
# A file with a partial last line (no trailing \n):
# {"id":1}\n{"id":2}\n{"id":3
# A correct parser returns records 1 and 2,
# and reports the partial line for the caller to handle.
# A parser that assumes every line is complete
# will fail on the last chunk.
# ============================================
# PART 4: THE ERROR POLICY
# ============================================
# Tolerant pipeline: skip bad lines, continue
# Strict pipeline: stop at first bad line, report line number
# The parser provides the line number; the caller decides.The parts covered incremental writing, constant-memory reading, the partial-line boundary, and the error-policy decision.
Quick Reference
Format Comparison
| Format | Separator | Enclosing | Newlines Inside |
|---|---|---|---|
| JSON | N/A | [] | Escaped only |
| JSON Lines | \n | None | Forbidden |
| NDJSON | \n | None | Forbidden |
| JSON Text Sequence | 0x1E | None | Allowed |
Streaming Parser Properties
| Property | Meaning |
|---|---|
values | Records parsed in this chunk |
read | Bytes/characters consumed |
done | Whether input is exhausted |
error | Parse failure, or null |
Error Policies
| Policy | Behavior |
|---|---|
| Tolerant | Skip bad line, continue |
| Strict | Stop at first bad line, report line number |
| Partial-preserving | Return valid records before error |
Large-File Processing
| Tool | Pattern |
|---|---|
| Python | for line in f: |
| Node.js | readline.createInterface |
| Go | bufio.Scanner with raised limit |
| Unix | jq -c, head, wc -l |
Best Practices
✅ Do This:
# Write one complete record per line
f.write(json.dumps(record) + "\n") # ✅
# Process line by line for constant memory
for line in f: process(json.loads(line)) # ✅
# Handle partial lines with a buffer and offset
buffer = buffer.slice(result.read) # ✅
# Decide the error policy explicitly
try: json.loads(line) except: skip_or_stop() # ✅❌ Don’t Do This:
# Append to a JSON array
f.write(",\n" + json.dumps(record)) # breaks the array # ❌
# Load a 1GB JSONL file into memory
data = [json.loads(l) for l in open("big.jsonl")] # ❌
# Assume every chunk ends on a line boundary
json.loads(chunk) # fails on partial last line # ❌
# Use a parser that discards prior records on error
# (loses valid data before the bad line) # ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| File not valid JSON | JSONL is not a JSON document | Parse per line |
| Last record missing | Partial line at chunk boundary | Buffer remainder, use read offset |
| Valid records lost on error | Parser throws and discards | Use per-line error handling |
| Newline inside string breaks line | Unescaped \n in JSON value | Ensure writer escapes newlines |
| Scanner buffer too small | Default 64KB limit | Raise to 16MB or use a reader |
Real-World Examples
1. Python line iteration
for line in open("data.jsonl"):
record = json.loads(line)2. Node.js readline
const rl = readline.createInterface({ input: createReadStream("data.jsonl") });3. Go decoder
dec := jsonl.NewDecoder(r)
for dec.More() { dec.Decode(&v) }4. Bun parseChunk
const { values, read } = Bun.JSONL.parseChunk(buffer);5. Error with line number
except json.JSONDecodeError as e:
print(f"Line {line_num}: {e}")6. Skip blank lines
if not line.strip(): continue7. Unix count
wc -l data.jsonl8. jq filter
jq -c 'select(.age > 30)' data.jsonl9. Append record
f.write(json.dumps(record) + "\n")10. Final partial line
if buffer: process(buffer)Visual
JSON Array vs JSON Lines
┌─────────────────────────────────────────────────────────┐
│ JSON ARRAY │
│ [ │
│ {"id":1}, │
│ {"id":2}, ◄── closing bracket ends the document │
│ {"id":3} │
│ ] │
│ Cannot append without rewriting │
└─────────────────────────────────────────────────────────┘┌─────────────────────────────────────────────────────────┐
│ JSON LINES │
│ {"id":1}\n │
│ {"id":2}\n ◄── each line is independent │
│ {"id":3}\n │
│ Append by adding a line │
└─────────────────────────────────────────────────────────┘Streaming Parse with Offset
┌─────────────────────────────────────────────────────────┐
│ CHUNK 1: {"a":1}\n{"b":2}\n{"c":3 │
│ │ │
│ ▼ │
│ parseChunk → values: [{a:1}, {b:2}] │
│ read: 16 (consumed) │
│ done: false (partial remains) │
│ │ │
│ ▼ │
│ buffer = chunk.slice(16) = '{"c":3' │
│ │ │
│ ▼ │
│ CHUNK 2: '}\n' │
│ buffer += chunk → '{"c":3}\n' │
│ │ │
│ ▼ │
│ parseChunk → values: [{c:3}] │
└─────────────────────────────────────────────────────────┘Error Policy Decision
┌─────────────────────────────────────────────────────────┐
│ LINE FAILS TO PARSE │
│ │ │
│ ┌────┴────┐ │
│ ▼ ▼ │
│ Tolerant Strict │
│ │ │ │
│ ▼ ▼ │
│ Skip, Stop, report │
│ continue line number │
│ │
│ Prior valid records: kept in both policies │
└─────────────────────────────────────────────────────────┘Large File Memory
┌─────────────────────────────────────────────────────────┐
│ LOAD ALL STREAM │
│ │
│ open(file).read() for line in open(file): │
│ → 3-5x file size in RAM → one line in RAM │
│ │
│ 1GB file → 3-5GB RAM 1GB file → constant │
│ May crash Processes 1GB in O(1) │
└─────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| JSON Lines / NDJSON | One JSON value per line, no brackets |
| Separator | \n or \r\n |
| Appendable | Yes, by adding a line |
| Valid JSON document | No, intentionally |
| Streaming parser offset | read property for chunk boundaries |
| Partial line | Buffer remainder, parse next chunk |
| Error policy | Tolerant (skip) or strict (stop) |
| Large files | Line iteration, constant memory |
Key takeaways:
- JSON Lines is JSON’s appendable form. Each line is a complete JSON value, and the absence of enclosing brackets means a writer can add a record without rewriting the file .
- The format is not valid JSON as a whole. It is a stream of JSON values, and a parser must process it line by line. Loading a 1GB JSONL file into a single array defeats the purpose and may exhaust memory .
- Streaming parsers must handle partial lines. A chunk from a network or file read may end mid-record. The parser reports how many bytes it consumed, and the caller buffers the remainder for the next chunk .
- Error handling is per-line, not per-document. A malformed line at position 500 does not invalidate the 499 records before it. The parser should preserve the valid records and report the line number, and the caller decides whether to skip or stop .
- JSON Text Sequences is the RFC alternative. RFC 7464 uses a record separator instead of a newline, allowing newlines within records at the cost of a control character that is awkward to edit .
Remember: JSON Lines exists because streams do not have a closing bracket. A log file, a data export, a message queue dump—these grow over time and are processed incrementally. The format’s rules are simple: one value per line, no unescaped newlines inside a value, and a parser that reports where it stopped. The failures come from assuming a line is complete when it is not, or discarding valid records because one line is malformed. Buffer the remainder, report the line number, and let the caller decide the error policy.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!