Rust 111 🦀 Advanced Macro Hygiene and TokenStream Manipulation - KandZ

⏱️ Estimated read: 12 min | |

Macro hygiene is the property that makes a macro’s internal identifiers invisible to the code that invokes it. A macro_rules! macro that declares a local variable x does not collide with the caller’s x, and a procedural macro that generates a function named helper does not overwrite a function the user already defined. The mechanism is the Span, a region of source code coupled with expansion information, and the rules that determine whether an identifier resolves at the macro’s definition site or at its call site. The TokenStream is the data structure that carries those spans, and quote is the library that constructs it.

This chapter covers the hygiene rules that apply to declarative and procedural macros, the Span constructors that control resolution, and the TokenStream manipulation patterns that the quote and proc-macro2 crates provide. It begins with the three transparency modes—opaque, semi-transparent, and transparent—and the mixed_site span that macro_rules! uses. It then explains quote‘s interpolation and hygiene behavior, the quote_spanned! macro for diagnostic spans, and the identifier construction patterns that avoid the concatenation pitfalls. The final sections cover the ToTokens trait and the testability that proc-macro2 provides.

Key point: Hygiene is determined by the span attached to each token, not by the macro’s name or its location. Span::call_site() resolves at the call site, Span::mixed_site() resolves locals at the definition site, and the default span from quote! is call_site.

Why hygiene is a span problem

The identifier collision problem. A macro that generates let x = 0; and then uses x in its expansion must not accidentally refer to a caller’s x. If the generated x resolved at the call site, the caller’s variable would be shadowed or the generated code would use the wrong value. Hygiene prevents this by giving the generated x a different “syntax context” from the caller’s x, so the two are distinct identifiers despite having the same name .

The three transparency modes. The compiler’s internal Transparency enum defines how an expansion resolves. Opaque means every identifier resolves at the definition site; this is the default for procedural macros. Transparent means every identifier resolves at the call site; this is what Span::call_site() produces. SemiTransparent is the macro_rules! behavior: local variables, labels, and $crate resolve at the definition site, but everything else resolves at the call site .

The mixed_site span. The Span::mixed_site() constructor produces exactly the semi-transparent behavior that macro_rules! uses. It is located at the call site (so error messages point to the invocation), but identifiers created with it resolve locals at the definition site . This is the span that a procedural macro should use when it wants to behave like a declarative macro.

The diagnostic span problem. A macro that generates an assertion on a user’s type should attach the user’s type span to the generated code, so that the error message points at the type the user wrote. quote_spanned! exists for this: it applies a given span to all tokens that originate within the quote, while preserving the spans of interpolated tokens . This is the difference between an error that says “the trait bound is not satisfied” with no location and one that highlights the exact type in the user’s source.

a. The Span constructors and their resolution rules

The proc_macro::Span type exposes three constructors that matter for hygiene: call_site, mixed_site, and the default span that quote! uses. The proc_macro2::Span type, which is the testable wrapper, exposes the same constructors.

Span::call_site() produces a span that resolves at the macro invocation site. Identifiers created with it are visible to the caller, and the caller’s code can refer to them. This is the span for generated identifiers that should be part of the public interface of the expansion—a generated function name that the caller is expected to use, or a variable that the caller passes in .

use proc_macro::Span;

// An identifier that resolves at the call site.
// The caller can refer to this identifier.
let span = Span::call_site();

Span::mixed_site() produces the macro_rules! behavior: locals, labels, and $crate resolve at the definition site, and everything else resolves at the call site . This is the correct choice for a procedural macro that wants to avoid capturing the caller’s local variables while still allowing the caller to refer to the generated items.

use proc_macro::Span;

// The macro_rules hygiene behavior.
// Locals resolve at the definition site; other names at the call site.
let span = Span::mixed_site();

The quote! macro uses Span::call_site() for tokens that originate within the invocation. Interpolated tokens preserve their own spans, which is why a variable passed into the quote retains its original hygiene . The quote_spanned! variant applies a given span to the originating tokens, which is the tool for diagnostic spans.

use quote::{quote, quote_spanned};

// Default: originating tokens get Span::call_site().
let tokens = quote! { let x = 0; };

// Spanned: originating tokens get the given span.
let span = /* the user's type span */;
let tokens = quote_spanned! { span => struct _AssertSync where #ty: Sync; };

b. `quote` interpolation and identifier construction

The quote! macro performs variable interpolation with #var, which is the equivalent of $var in macro_rules!. Any type that implements ToTokens can be interpolated, which includes the syn AST types and the primitive types . Repetition uses #(...)* or #(...),*, and the repetition variable can be a Vec, a slice, a BTreeSet, or any iterator .

use quote::quote;

// Interpolation: #name inserts the value of `name`.
let name = /* ... */;
let tokens = quote! {
    fn #name() {}
};

// Repetition: #(#field)* iterates over the fields.
let fields = vec![/* ... */];
let tokens = quote! {
    #( struct #fields; )*
};

The identifier concatenation pitfall is the most common mistake. Interpolating an identifier next to an underscore produces two separate tokens, _ and the identifier, not a single concatenated identifier. The format_ident! macro is the correct tool: it creates a new Ident with the concatenated name and the specified span .

use quote::{quote, format_ident};

// Incorrect: produces `_` and `x` as separate tokens.
// let tokens = quote! { let mut _#ident = 0; };

// Correct: produces a single identifier `_x`.
let varname = format_ident!("_{}", ident);
let tokens = quote! { let mut #varname = 0; };

The ToTokens trait is the abstraction that makes interpolation work. It has a to_tokens method that writes the value into a TokenStream, and a to_token_stream convenience method. Implementing it for a custom type allows that type to be interpolated into quote! . The trait is sealed in some versions of the library, which means it cannot be implemented for types outside the crate that defines it in those versions.

c. TokenStream manipulation and testability

The proc_macro2::TokenStream is the testable representation. The proc_macro::TokenStream type can only exist inside a procedural macro, which makes it impossible to unit test the token manipulation logic outside of a compiler invocation. The proc_macro2 crate provides a compatible TokenStream that can be created in a test, which is why the ecosystem is built around it . The conversion between the two is a From implementation in both directions, and the typical pattern is tokens.into() at the end of the macro.

use proc_macro2::TokenStream;
use quote::quote;

// The testable version.
let tokens: TokenStream = quote! { fn foo() {} };

// In the procedural macro, convert to the compiler's type.
// let compiler_tokens: proc_macro::TokenStream = tokens.into();

The quote macro produces a proc_macro2::TokenStream, and the quote_spanned! macro does the same with the given span. The TokenStream can be built incrementally with the TokenStreamExt trait, which provides append methods for Ident, Punct, Literal, and other token types. This is the manual construction path for cases where quote! is not expressive enough.

Complete Example Session

// ============================================
// PART 1: THE HYGIENE PROBLEM
// ============================================
// A macro that generates `let x = 0;` must not
// collide with the caller's `x`.

// macro_rules! make_x {
//     () => { let x = 0; };
// }
// let x = 5;
// make_x!();
// println!("{}", x); // prints 5, not 0

// ============================================
// PART 2: THE TRANSPARENCY MODES
// ============================================
// Opaque: resolves at definition site (proc macro default).
// SemiTransparent: locals def-site, rest call-site (macro_rules).
// Transparent: all call-site (Span::call_site).

// ============================================
// PART 3: THE SPAN CONSTRUCTORS
// ============================================
use proc_macro::Span;

let call = Span::call_site();    // transparent
let mixed = Span::mixed_site();  // semi-transparent

// ============================================
// PART 4: THE QUOTE DEFAULT
// ============================================
// Tokens originating in quote! get call_site().
// Interpolated tokens keep their own spans.

use quote::quote;

let ident = /* ... */;
let tokens = quote! {
    fn #ident() {}  // `fn` is call_site, `ident` keeps its span
};

// ============================================
// PART 5: THE DIAGNOSTIC SPAN
// ============================================
// quote_spanned! applies a given span to originating tokens.
// Used to point errors at the user's code.

use quote::quote_spanned;

let ty_span = /* the span of the user's type */;
let assert_sync = quote_spanned! { ty_span =>
    struct _AssertSync where #ty: Sync;
};

// ============================================
// PART 6: THE IDENTIFIER CONCATENATION
// ============================================
// Use format_ident! to build a new identifier.

use quote::format_ident;

let varname = format_ident!("_{}", ident);
let tokens = quote! { let mut #varname = 0; };

// ============================================
// PART 7: THE ToTokens TRAIT
// ============================================
// Any type implementing ToTokens can be interpolated.

// impl ToTokens for MyType {
//     fn to_tokens(&self, tokens: &mut TokenStream) { ... }
// }

// ============================================
// PART 8: THE PROC_MACRO2 TESTABILITY
// ============================================
// proc_macro2::TokenStream can be created in tests.
// proc_macro::TokenStream cannot.

use proc_macro2::TokenStream as TokenStream2;

let tokens: TokenStream2 = quote! { fn foo() {} };
// In the macro: proc_macro::TokenStream::from(tokens)

// ============================================
// PART 9: THE MANUAL CONSTRUCTION
// ============================================
// TokenStreamExt provides append methods.

use quote::TokenStreamExt;
// let mut tokens = TokenStream2::new();
// tokens.append(ident);
// tokens.append(Punct::new(';', Spacing::Alone));

// ============================================
// PART 10: THE FULL PICTURE
// ============================================
// Span determines resolution.
// quote! defaults to call_site.
// quote_spanned! sets the diagnostic span.
// format_ident! builds identifiers correctly.

The ten parts build a conceptual view of the hygiene system: the problem, the transparency modes, the span constructors, the quote default, the diagnostic span, the identifier construction, the ToTokens trait, the testability, and the manual construction. The pattern is consistent: the span is the unit of hygiene, and the TokenStream is the container.

Quick Reference

Span Constructors

Constructor	Resolution	Use Case
`Span::call_site()`	Call site	Public generated names
`Span::mixed_site()`	`macro_rules!` behavior	Locals def-site, rest call-site
`quote!` default	`call_site()`	Originating tokens
`quote_spanned!(s=>)`	Given span `s`	Diagnostic spans

Transparency Modes

Mode	Locals	Labels	`$crate`	Other
`Opaque`	Def-site	Def-site	Def-site	Def-site
`SemiTransparent`	Def-site	Def-site	Def-site	Call-site
`Transparent`	Call-site	Call-site	Call-site	Call-site

quote! Interpolation

Syntax	Meaning
`#var`	Insert variable
`#(#var)*`	Repeat without separator
`#(#var),*`	Repeat with comma
`format_ident!("_{}", x)`	Build identifier

Testability

Type	Testable	Context
`proc_macro::TokenStream`	No	Inside macro only
`proc_macro2::TokenStream`	Yes	Anywhere

Best Practices

✅ Do This:

// Use mixed_site for macro_rules-like hygiene.
let span = Span::mixed_site();                                        // ✅

// Use quote_spanned! for diagnostic spans.
quote_spanned! { ty_span => struct _Assert where #ty: Sync; }         // ✅

// Use format_ident! for concatenated identifiers.
let varname = format_ident!("_{}", ident);                            // ✅

// Build the TokenStream with proc_macro2 for testability.
let tokens: proc_macro2::TokenStream = quote! { ... };                // ✅

❌ Don’t Do This:

// Don't interpolate identifiers next to underscores.
quote! { let mut _#ident = 0; }  // two tokens, not one               // ❌

// Don't use call_site for locals that must not be captured.
// The caller's variable may be captured.                            // ❌

// Don't construct proc_macro::TokenStream in tests.
// It panics outside a macro invocation.                             // ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
Caller’s variable captured	`call_site` span on a local	`mixed_site`
Error points at macro, not user	`quote!` default span	`quote_spanned!` with user span
`_x` not concatenated	Interpolation creates separate tokens	`format_ident!`
Test cannot create tokens	`proc_macro` type is macro-only	`proc_macro2`

Real-World Examples

1. Call Site

Span::call_site()

2. Mixed Site

Span::mixed_site()

3. Quote Default

quote! { fn foo() {} }

4. Quote Spanned

quote_spanned! { ty.span() => struct _Assert where #ty: Sync; }

5. Format Ident

format_ident!("_{}", ident)

6. ToTokens

impl ToTokens for MyType { fn to_tokens(&self, t: &mut TokenStream) { ... } }

7. proc_macro2

let tokens: proc_macro2::TokenStream = quote! { ... };

8. Conversion

proc_macro::TokenStream::from(tokens)

9. Repetition

#(#field)*

10. Append

tokens.append(ident)

Visual

The Span Resolution

┌─────────────────────────────────────────────────────────┐
│  MACRO DEFINITION SITE                                  │
│  fn my_macro() {                                        │
│      let x = 0;       ← def-site x                      │
│  }                                                      │
│                                                         │
│  MACRO CALL SITE                                        │
│  fn caller() {                                          │
│      let x = 5;       ← call-site x                     │
│      my_macro!();                                       │
│  }                                                      │
│                                                         │
│  OPAQUE: expansion's x resolves at def-site.            │
│  TRANSPARENT: expansion's x resolves at call-site.      │
│  SEMI-TRANSPARENT: locals at def-site, rest call-site.  │
└─────────────────────────────────────────────────────────┘

quote! Hygiene

┌─────────────────────────────────────────────────────────┐
│  let tokens = quote! {                                  │
│      fn #ident() {}                                     │
│  };                                                     │
│                                                         │
│  Originating tokens (`fn`, `()`, `{}`, `;`):            │
│    → Span::call_site()                                  │
│                                                         │
│  Interpolated token (#ident):                           │
│    → the span that `ident` already has                  │
│                                                         │
│  quote_spanned! { span => ... }:                        │
│    → originating tokens get `span`                      │
└─────────────────────────────────────────────────────────┘

The Identifier Pitfall

┌─────────────────────────────────────────────────────────┐
│  INCORRECT                                              │
│  quote! { let mut _#ident = 0; }                        │
│         │                                               │
│         ▼  two tokens                                   │
│  `_`  and  `x`  →  `_ x`, not `_x`                      │
│                                                         │
│  CORRECT                                                │
│  let varname = format_ident!("_{}", ident);             │
│  quote! { let mut #varname = 0; }                       │
│         │                                               │
│         ▼  one token                                    │
│  `_x`                                                   │
└─────────────────────────────────────────────────────────┘

proc_macro vs proc_macro2

┌─────────────────────────────────────────────────────────┐
│  proc_macro::TokenStream                                │
│  → only inside a procedural macro                       │
│  → cannot be tested                                     │
│                                                         │
│  proc_macro2::TokenStream                               │
│  → anywhere (tests, main.rs, build.rs)                  │
│  → testable                                             │
│  → From conversion to proc_macro                        │
└─────────────────────────────────────────────────────────┘

Summary

Item	Value
Hygiene unit	`Span`
Transparent span	`Span::call_site()`
Macro_rules span	`Span::mixed_site()`
Diagnostic span	`quote_spanned!`
Identifier construction	`format_ident!`
Testable tokens	`proc_macro2::TokenStream`
Interpolation	`#var`
Repetition	`#(...)*`

Key takeaways:

Hygiene is determined by the span attached to each token. call_site resolves at the invocation, mixed_site resolves locals at the definition site, and the default quote! span is call_site .
macro_rules! uses semi-transparent hygiene. Local variables, labels, and $crate resolve at the definition site; everything else resolves at the call site .
quote_spanned! is for diagnostic spans. It applies a given span to the tokens that originate within the quote, so that errors point at the user’s code rather than the macro .
format_ident! is the identifier constructor. Interpolating an identifier next to an underscore produces two tokens, not a concatenated one. The format_ident! macro builds a new identifier with the correct name and span .
proc_macro2 is the testable representation. proc_macro::TokenStream cannot exist outside a macro invocation. proc_macro2::TokenStream can be created anywhere, which is why the ecosystem is built around it .
ToTokens is the interpolation trait. Any type that implements it can be interpolated into quote!. The syn AST types implement it, and custom types can implement it for their own expansion logic .

Remember: Hygiene is not a property of the macro; it is a property of the span on each token. A macro that generates a local variable with call_site will capture the caller’s variable. A macro that generates a diagnostic with the wrong span will produce an error that points at the macro instead of the user’s code. The Span constructors and the quote macros are the tools for controlling this, and the format_ident! macro is the tool for building identifiers that cannot be built by interpolation alone.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!