Serialization 17 📄 Cross-Format Conversion — Tooling and Pitfalls - KandZ

⏱️ Estimated read: 11 min | |

Cross-format conversion is not a neutral translation. Each format has a data model with different primitives, different structural guarantees, and different opinions about what a value means. JSON has null; TOML does not. XML has attributes and namespaces; JSON has neither. YAML has anchors and tags; JSON has no representation for them. A converter that ignores these differences produces output that parses successfully but means something different from the input. A converter that tries to preserve everything produces output that is not valid in the target format.

Quick How-To Guides & Fast FixesNeed a quick fix or step-by-step solution? Browse our fast answers on HowTo.KandZ.me:

Operating Systems CLI →Dev Environment Languages →Web Network Tools →Security Accounts →Security Access Tech →Device Phone Basics →Social Messaging →Digital Files Media →

This chapter covers the conversion tooling that works, the asymmetries that cause silent data loss, and the pitfalls that make a round trip fail to return what it took. The sections that follow walk through the tools, the structural mismatches, and the conversion patterns that preserve meaning rather than just syntax.

Key point: Conversion is lossy whenever the source format has a feature the target format cannot express; TOML’s lack of null, XML’s attributes and namespaces, and YAML’s anchors and tags are the three asymmetries that cause the most silent data loss.

a. The tooling landscape

The conversion tools fall into three categories: general-purpose processors that handle multiple formats, single-purpose converters that do one direction, and libraries that expose conversion as an API.

yq (Go, Mike Farah) is the most capable general-purpose tool. It auto-detects input format from the extension, supports -p/--input-format and -o/--output-format for explicit control, and handles YAML, JSON, XML, CSV, TOML, HCL, properties, and more . The -P flag produces idiomatic YAML from JSON, and -oy sets the output to YAML . The same tool converts TOML to JSON and JSON to TOML, which makes it the default choice for one-off conversions.

yq (Python, Andrey Kislyuk) is a different tool with a different design. It converts YAML to JSON, pipes the result to jq, and optionally converts back to YAML with -y. The -Y flag enables roundtrip mode, which preserves YAML tags and styles by injecting metadata into the JSON representation . The trade-off is that the JSON is no longer plain data; a jq filter that counts array entries will count the metadata keys as well, and the tool’s documentation warns that -Y is incompatible with filters that do not expect the extra information .

remarshal is a focused converter for CBOR, JSON, MessagePack, TOML, and YAML. It documents the format limitations explicitly: binary fields cannot be converted to JSON or TOML; TOML’s local time cannot be converted to a date-time in another format; date-time types fail conversion to JSON without the -k/--stringify option . The --wrap and --unwrap options solve the problem of converting non-dictionary data to TOML, which requires a top-level dictionary and rejects top-level arrays .

dataconv and confease are Python libraries that expose load, save, and convert functions with format auto-detection. confease documents the lossy boundary directly: “TOML has no null value, and JSON does not support native Python dates… Unsupported values fail saving before replacement; values are not silently stringified” . The failure-before-replacement behavior is the correct default; a converter that silently stringifies a null into the string "null" has changed the data’s type without telling anyone.

b. The structural asymmetries

The conversion failures cluster around three format features that have no equivalent in the others.

TOML’s null gap. TOML has no null value. A JSON document with {"key": null} cannot be represented in TOML directly. confease fails the save rather than inventing a TOML null marker . The alternatives are to omit the key, to use an empty string (which is a string, not a null), or to wrap the document so the null is represented as a sentinel value that the reader understands. None of these is a conversion; each is a data-model decision the application must make.

XML’s attributes and namespaces. XML has attributes (<element attr="value">), namespaces (xmlns:prefix), and mixed content (text interleaved with elements). JSON has none of these. The conversion conventions differ: the Parker convention is lossy and lean, ideal for simple record-like XML where attributes and namespaces are not needed. Badgerfish and GData preserve attributes (prefixed with @) and namespaces (in @xmlns properties) at the cost of a more complex JSON structure . A converter that uses Parker on a document with attributes silently discards them; the choice of convention determines what survives.

YAML’s anchors, tags, and comments. JSON has no anchors, no tags, and no comments. A YAML document with <<: *defaults converts to JSON with the merged object inlined; the anchor and merge key are gone. A YAML document with !env DATABASE_URL either fails to parse or produces a value that has lost the tag. Comments are discarded entirely. The Python yq wrapper’s -Y flag preserves tags and styles via metadata, but not comments, and the metadata breaks plain JSON consumers .

c. The round-trip failure and the YAML 1.1 trap

A conversion from A to B and back to A is not guaranteed to produce the original document. The failures are structural, not accidental.

The YAML 1.1 boolean problem. A YAML file with enabled: yes parses to the boolean true under a 1.1 parser. Converting to JSON produces {"enabled": true}. Converting back to YAML produces enabled: true, not enabled: yes. The value is the same; the bytes are not. If the original file’s yes was intended as a string—which it is under YAML 1.2—the round trip through a 1.1 parser has changed the type .

The TOML formatting loss. TOML.stringify(JSON.parse(JSON.stringify(TOML.parse(toml)))) does not produce the original TOML. The j-toml library preserves the original formatting when the object is passed directly from parse to stringify, but a “purification” through JSON serialization removes the formatting metadata . The arrays are reformatted, and the tables are reordered. The data is the same; the document is not.

The XML attribute loss. Converting <element attr="value">text</element> to JSON with the Parker convention produces {"element": "text"}. The attribute is gone. Converting back to XML produces <element>text</element>, which is a different document. The Badgerfish convention preserves the attribute as {"element": {"@attr": "value", "$": "text"}}, but the round trip through Badgerfish and back is not necessarily byte-identical either .

The round-trip pattern that works is the one that treats the conversion as a one-way projection. If the target format cannot express a feature, the conversion drops it, and the source format remains the source of truth. If the target format must round-trip, the converter needs a preservation channel—metadata keys, sidecar files, or a format that is a superset of both (which is what confease and omnist aim to provide ).

Complete Example Session

# ============================================
# PART 1: JSON TO YAML WITH yq
# ============================================
# echo '{"name":"app","version":"1.10","enabled":true}' | yq -P -oy
# name: app
# version: "1.10"     # quoted to preserve string
# enabled: true

# ============================================
# PART 2: TOML TO JSON WITH yq
# ============================================
# yq -p toml -o json pyproject.toml
# {"project":{"name":"my-package","version":"1.0.0"}}

# ============================================
# PART 3: JSON TO TOML (WRAPPING REQUIRED)
# ============================================
# echo '[{"a":"b"}]' | remarshal --from json --to toml
# Error: cannot convert non-dictionary data to TOML; use "--wrap"

# echo '[{"a":"b"}]' | remarshal --from json --to toml --wrap data
# [[data]]
# a = "b"

# ============================================
# PART 4: THE NULL GAP
# ============================================
# JSON: {"key": null}
# TOML: no null value exists
# confease: fails saving rather than stringifying
# The application must decide: omit, sentinel, or restructure

# ============================================
# PART 5: XML TO JSON CONVENTIONS
# ============================================
# XML: <element attr="value">text</element>
# Parker:     {"element": "text"}           (attribute lost)
# Badgerfish: {"element": {"@attr": "value", "$": "text"}}  (preserved)

The parts covered JSON-to-YAML, TOML-to-JSON, the TOML top-level dictionary requirement, the null gap, and the XML convention choice.

Quick Reference

Conversion Tools

Tool	Formats	Notes
`yq` (Go)	YAML, JSON, XML, TOML, CSV, HCL	Auto-detect, `-p`/`-o` flags
`yq` (Python)	YAML ↔ JSON via `jq`	`-Y` roundtrip with metadata
`remarshal`	CBOR, JSON, MessagePack, TOML, YAML	Documents format limits
`dataconv`	JSON, YAML, TOML, CSV	`load`/`save`/`convert` API
`confease`	YAML, JSON, TOML, XML, INI, CSV	Fails on unsupported values

Format Asymmetries

Feature	Source	Target Gap
Null	JSON	TOML has no null
Attributes	XML	JSON has no attributes
Namespaces	XML	JSON has no namespaces
Anchors/aliases	YAML	JSON has no references
Tags	YAML	JSON has no tags
Comments	YAML/TOML	JSON has no comments
Binary	CBOR/MessagePack	JSON/TOML cannot represent

XML-to-JSON Conventions

Convention	Attributes	Namespaces	Use For
Parker	Lost	Lost	Simple records
Badgerfish	`@attr`	`@xmlns`	Full preservation
GData	`$t`, `@`	`@xmlns`	Google APIs

Best Practices

✅ Do This:

# Use yq for multi-format conversion
yq -p toml -o json config.toml                             # ✅

# Wrap non-dictionary data for TOML
remarshal --from json --to toml --wrap data                # ✅ 

# Choose the XML convention explicitly
# Badgerfish when attributes matter                         # ✅

# Accept that the source format remains the source of truth
# Do not expect a round trip to return the original         # ✅

❌ Don’t Do This:

# Convert YAML with anchors to JSON and expect to get them back
# (anchors are resolved)                                     # ❌

# Convert JSON null to TOML
# (no null value exists)                                     # ❌

# Use Parker on XML with attributes
# (attributes are silently lost)                             # ❌

# Assume TOML stringify after JSON purification preserves format
# (formatting metadata is gone)                              # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Null becomes string	TOML has no null	Omit, sentinel, or fail
Attributes lost in XML→JSON	Parker convention	Use Badgerfish
YAML `yes` becomes `true` in round trip	1.1 parser resolves boolean	Quote or use 1.2
TOML formatting changes	Formatting metadata lost through JSON	Use format-preserving library
`yq -Y` breaks `jq` filters	Metadata keys injected	Do not use `-Y` for plain data
Top-level array fails TOML	TOML requires dictionary	Use `--wrap`

Real-World Examples

1. JSON to YAML

yq -P -oy input.json

2. TOML to JSON

yq -p toml -o json config.toml

3. TOML with wrapping

remarshal --from json --to toml --wrap data

4. XML with Badgerfish

{"element": {"@attr": "value", "$": "text"}}

5. YAML roundtrip tags

yq -Y '.Resources' template.yaml

6. Null gap failure

confease: fails saving null to TOML

7. Datetime to JSON

remarshal --from toml --to json -k config.toml

8. Format auto-detection

dataconv convert data.json data.yaml

9. YAML 1.1 boolean roundtrip

enabled: yes → true → enabled: true

10. TOML formatting loss

TOML.parse → JSON → JSON.parse → TOML.stringify
# formatting metadata is gone

Visual

The Conversion Pipeline

┌─────────────────────────────────────────────────────────┐
│  SOURCE FORMAT                                          │
│         │                                               │
│         ▼                                               │
│  Parse to canonical model                               │
│  (dict, list, scalars)                                  │
│         │                                               │
│         ▼                                               │
│  Serialize to target format                             │
│  (lossy: features without equivalent are dropped)       │
│         │                                               │
│         ▼                                               │
│  TARGET FORMAT                                          │
└─────────────────────────────────────────────────────────┘

The Three Asymmetries

┌─────────────────────────────────────────────────────────┐
│  NULL GAP (JSON → TOML)                                 │
│  {"key": null} → no TOML equivalent                     │
│  Options: omit, sentinel, fail                          │
│                                                         │
│  ATTRIBUTE GAP (XML → JSON)                             │
│  <el attr="v">text</el> → Parker loses attr             │
│  Badgerfish preserves as @attr                          │
│                                                         │
│  REFERENCE GAP (YAML → JSON)                            │
│  <<: *defaults → merged, anchor gone                    │
│  !tag value → lost or error                             │
└─────────────────────────────────────────────────────────┘

XML Conventions

┌─────────────────────────────────────────────────────────┐
│  PARKER                    BADGERFISH                   │
│                                                         │
│  {"el": "text"}            {"el": {"@attr": "v",        │
│  attributes lost                  "$": "text"}}         │
│  namespaces lost           attributes preserved         │
│                            namespaces in @xmlns         │
│                                                         │
│  Simple records            Full preservation            │
│  Book lists                ATOM feeds, SOAP             │
└─────────────────────────────────────────────────────────┘

Round-Trip Failure

┌─────────────────────────────────────────────────────────┐
│  A → B → A                                              │
│                                                         │
│  Data:      same                                        │
│  Bytes:     different                                   │
│  Type:      may change (YAML yes → true)                │
│  Formatting: lost (TOML arrays, YAML comments)          │
│                                                         │
│  The source format remains the source of truth          │
└─────────────────────────────────────────────────────────┘

Summary

Item	Value
General tool	`yq` (Go) with `-p`/`-o`
YAML roundtrip	`yq` (Python) `-Y` with metadata
Documented limits	`remarshal` (binary, dates, null)
TOML null	No null value; fail or omit
XML attributes	Parker loses, Badgerfish preserves
YAML anchors	Resolved to inline data in JSON
Top-level array to TOML	Requires `--wrap`
Round trip	Data preserved, document not

Key takeaways:

Conversion is lossy when the target format cannot express a source feature. TOML has no null, JSON has no attributes or namespaces, and JSON has no anchors, tags, or comments. The converter’s job is to fail or document the loss, not to silently invent a representation .
The tool choice determines the loss. yq (Go) auto-detects and handles the most formats; remarshal documents its limits explicitly; yq (Python) -Y preserves YAML tags via metadata at the cost of non-standard JSON . Choose the tool whose loss profile matches the requirement.
XML-to-JSON requires a convention decision. Parker is lean and lossy; Badgerfish preserves attributes and namespaces. The choice is the application’s, and the converter should expose it rather than defaulting silently .
TOML’s top-level dictionary requirement is a hard constraint. A JSON array cannot become TOML without wrapping it in a dictionary. remarshal --wrap is the documented pattern .
A round trip does not return the original. The data is preserved; the document is not. Formatting, comments, anchors, and type spellings are lost or normalized. The source format remains the source of truth, and the converted format is a projection for its consumers.

Remember: Cross-format conversion is a projection from a richer data model to a poorer one, or between two models with different primitives. The question is not “does the output parse?” but “what did the conversion drop, and did it drop it silently?” Use a tool that documents its limits, choose the XML convention deliberately, wrap top-level arrays for TOML, and accept that the source format is the authority. When a round trip must be exact, do not convert—keep the source and derive the projection on demand.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!