Dart 97 🎯 Building a Custom Transpiler Engine or DSL Parser in Pure Dart
A custom transpiler or DSL parser in Dart is not a single program but a pipeline of three cooperating components: a lexer that turns source text into tokens, a parser that turns tokens into a syntax tree, and a code generator or evaluator that walks the tree and produces output. The architecture is the same whether the target is a configuration language, a query DSL, or a source-to-source transpiler. The difference is in what the tree walk does: a transpiler emits code for another language, and a DSL parser evaluates or interprets the tree directly.
Key point: The parser is the heart of the pipeline, and the choice of parsing strategy determines the complexity. A recursive-descent parser hand-written in Dart is the most direct approach and the easiest to debug. A parser-combinator library like PetitParser trades some performance for composability and testability. A grammar-driven generator like smart_parser trades flexibility for the ability to define the grammar in a separate file and generate the parser code.
Why pure Dart is a viable host for parsers
The package ecosystem problem. A parser does not need a runtime written in C or a compiler toolchain. It needs string manipulation, a token stream, and a tree structure, all of which Dart provides natively. The petitparser package is the evidence: it implements grammars for JSON, Pascal, Python, Smalltalk, Dart itself, and a Lisp interpreter, entirely in plain Dart . The package is type-safe, composable, and has no native dependencies.
The tooling problem. Dart’s tooling supports parser development directly. The slex package provides a configurable lexer with indentation-based scanning, C-like scanning, and configurable comment and number handling . The smart_parser package generates a parser from a grammar file written in a section-based DSL, with the generator itself written in Dart . A developer can write a grammar, run the generator, and get a Dart parser class without leaving the Dart ecosystem.
The transpiler problem. A transpiler is a parser plus a code generator. The Dart team’s own dart2js compiler uses an SSA-based intermediate representation and a JavaScript emitter, but that is a production compiler with years of optimization work . A custom transpiler for a smaller language does not need that complexity. The pipeline is: lexer → parser → AST → emitter. Each stage is a Dart program, and the AST is a set of Dart classes. The krom_script package demonstrates this: it includes a Lexer, a Parser, an ASTPrinter that converts the AST back to source code, and an Interpreter that evaluates it .
a. The Lexer: Tokenizing the Source
The lexer, also called a scanner, reads the source string character by character and emits a stream of tokens. Each token has a type, a value, and a position. The position is what allows the parser to report errors with line and column information.
A hand-written lexer in Dart is a class with a position field and methods that advance through the input. The slex package provides this as a configurable library: the developer sets the comment syntax, the terminal operators, and the number formats, and the lexer handles the rest . A minimal hand-written lexer for a simple DSL might look like this:
class Token {
final String type;
final String value;
final int position;
Token(this.type, this.value, this.position);
}
class Lexer {
final String source;
int _pos = 0;
Lexer(this.source);
List<Token> tokenize() {
final tokens = <Token>[];
while (_pos < source.length) {
final ch = source[_pos];
if (ch == ' ' || ch == '\n' || ch == '\t') { _pos++; continue; }
if (RegExp(r'[a-zA-Z_]').hasMatch(ch)) {
final start = _pos;
while (_pos < source.length && RegExp(r'[a-zA-Z0-9_]').hasMatch(source[_pos])) { _pos++; }
tokens.add(Token('IDENT', source.substring(start, _pos), start));
} else if (RegExp(r'[0-9]').hasMatch(ch)) {
final start = _pos;
while (_pos < source.length && RegExp(r'[0-9]').hasMatch(source[_pos])) { _pos++; }
tokens.add(Token('NUMBER', source.substring(start, _pos), start));
} else {
tokens.add(Token('TERMINAL', ch, _pos));
_pos++;
}
}
return tokens;
}
}The lexer does not validate the sequence of tokens. A number followed by an identifier followed by a terminal is a valid token stream even if it is not a valid program. The parser is responsible for checking the grammar.
b. The Parser: Building the Syntax Tree
The parser consumes the token stream and produces an abstract syntax tree. The tree is a set of Dart classes, one per grammar production. For a calculator DSL, the tree nodes might be NumberLiteral, BinaryExpression, and Variable. For a transpiler, the nodes correspond to the constructs of the source language.
A recursive-descent parser is a direct translation of the grammar into Dart methods. Each method corresponds to a production rule and calls the methods for the rules it references.
abstract class Expr {}
class NumberLiteral extends Expr {
final num value;
NumberLiteral(this.value);
}
class BinaryExpr extends Expr {
final Expr left;
final String op;
final Expr right;
BinaryExpr(this.left, this.op, this.right);
}
class Parser {
final List<Token> tokens;
int _pos = 0;
Parser(this.tokens);
Expr parseExpression() {
var left = parseTerm();
while (_peek().value == '+' || _peek().value == '-') {
final op = _advance().value;
final right = parseTerm();
left = BinaryExpr(left, op, right);
}
return left;
}
Expr parseTerm() {
var left = parseFactor();
while (_peek().value == '*' || _peek().value == '/') {
final op = _advance().value;
final right = parseFactor();
left = BinaryExpr(left, op, right);
}
return left;
}
Expr parseFactor() {
final token = _advance();
if (token.type == 'NUMBER') return NumberLiteral(num.parse(token.value));
throw FormatException('Unexpected token ${token.value}');
}
Token _peek() => _pos < tokens.length ? tokens[_pos] : Token('EOF', '', -1);
Token _advance() => tokens[_pos++];
}The petitparser package provides a combinator approach that is more composable than hand-written recursive descent. A grammar is built from parser objects that are combined with operators: & for sequence, | for choice, star() and plus() for repetition, and .map() for attaching actions . The ExpressionBuilder in the same package handles operator precedence and associativity, which is the part of a parser that is most error-prone to write by hand .
The smart_parser package takes a third approach: the grammar is written in a separate file with a section-based syntax, and the ParserGenerator reads the grammar and emits a Dart parser class . The generator supports PEG expressions like sequences, choices, and predicates, and it allows error handlers to be attached to grammar rules with the ~{} notation .
c. The Backend: Evaluation or Code Generation
The backend walks the AST and produces the output. For a DSL parser, the backend is an evaluator. For a transpiler, the backend is a code generator that emits target-language source.
An evaluator is a visitor over the AST classes. Each node type has a corresponding visit method that returns a value.
class Evaluator {
final Map<String, num> variables;
Evaluator(this.variables);
num evaluate(Expr expr) {
if (expr is NumberLiteral) return expr.value;
if (expr is BinaryExpr) {
final left = evaluate(expr.left);
final right = evaluate(expr.right);
switch (expr.op) {
case '+': return left + right;
case '-': return left - right;
case '*': return left * right;
case '/': return left / right;
}
}
throw StateError('Unknown expression');
}
}A code generator has the same structure but returns a string instead of a number. For a transpiler, the generator might emit Dart code from a DSL, or JavaScript from a Dart subset. The krom_script package includes an ASTPrinter that converts the AST back to KromScript source, which is the simplest form of code generation .
The boundary between a transpiler and an interpreter is the boundary between emitting a string and returning a value. Both walk the same AST. The choice depends on whether the output is meant to be executed by another runtime or evaluated in place.
Complete Example Session
// ============================================
// PART 1: TOKEN DEFINITION
// ============================================
class Token {
final String type;
final String value;
final int position;
Token(this.type, this.value, this.position);
}
// ============================================
// PART 2: LEXER
// ============================================
class Lexer {
final String source;
int _pos = 0;
Lexer(this.source);
List<Token> tokenize() {
final tokens = <Token>[];
while (_pos < source.length) {
final ch = source[_pos];
if (ch.trim().isEmpty) { _pos++; continue; }
if (RegExp(r'[a-zA-Z_]').hasMatch(ch)) {
final start = _pos;
while (_pos < source.length && RegExp(r'[a-zA-Z0-9_]').hasMatch(source[_pos])) { _pos++; }
tokens.add(Token('IDENT', source.substring(start, _pos), start));
} else if (RegExp(r'[0-9]').hasMatch(ch)) {
final start = _pos;
while (_pos < source.length && RegExp(r'[0-9]').hasMatch(source[_pos])) { _pos++; }
tokens.add(Token('NUMBER', source.substring(start, _pos), start));
} else {
tokens.add(Token('TERMINAL', ch, _pos));
_pos++;
}
}
return tokens;
}
}
// ============================================
// PART 3: AST NODES
// ============================================
abstract class Expr {}
class NumberLiteral extends Expr {
final num value;
NumberLiteral(this.value);
}
class BinaryExpr extends Expr {
final Expr left;
final String op;
final Expr right;
BinaryExpr(this.left, this.op, this.right);
}
// ============================================
// PART 4: PARSER — RECURSIVE DESCENT
// ============================================
class Parser {
final List<Token> tokens;
int _pos = 0;
Parser(this.tokens);
Expr parseExpression() {
var left = parseTerm();
while (_peek().value == '+' || _peek().value == '-') {
final op = _advance().value;
left = BinaryExpr(left, op, parseTerm());
}
return left;
}
Expr parseTerm() {
var left = parseFactor();
while (_peek().value == '*' || _peek().value == '/') {
final op = _advance().value;
left = BinaryExpr(left, op, parseFactor());
}
return left;
}
Expr parseFactor() {
final token = _advance();
if (token.type == 'NUMBER') return NumberLiteral(num.parse(token.value));
throw FormatException('Unexpected token ${token.value} at ${token.position}');
}
Token _peek() => _pos < tokens.length ? tokens[_pos] : Token('EOF', '', -1);
Token _advance() => tokens[_pos++];
}
// ============================================
// PART 5: EVALUATOR
// ============================================
class Evaluator {
num evaluate(Expr expr) {
if (expr is NumberLiteral) return expr.value;
if (expr is BinaryExpr) {
final left = evaluate(expr.left);
final right = evaluate(expr.right);
switch (expr.op) {
case '+': return left + right;
case '-': return left - right;
case '*': return left * right;
case '/': return left / right;
}
}
throw StateError('Unknown expression');
}
}
// ============================================
// PART 6: WIRE THE PIPELINE
// ============================================
void main() {
final source = '2 + 3 * 4';
final tokens = Lexer(source).tokenize();
final ast = Parser(tokens).parseExpression();
final result = Evaluator().evaluate(ast);
print(result); // 14
}
// ============================================
// PART 7: USING PETITPARSER FOR THE SAME GRAMMAR
// ============================================
// import 'package:petitparser/petitparser.dart';
//
// final builder = ExpressionBuilder<num>();
// builder.primitive(digit().plus().flatten().trim().map(num.parse));
// builder.group().left(char('*').trim(), (a, op, b) => a * b);
// builder.group().left(char('+').trim(), (a, op, b) => a + b);
// final parser = builder.build().end();
// print(parser.parse('2 + 3 * 4').value); // 14
// ============================================
// PART 8: USING SMART_PARSER FOR GRAMMAR GENERATION
// ============================================
// Write a .grammar file with sections and rules
// Run ParserGenerator to produce a parser class
// The generated class is a Dart file that can be imported
// ============================================
// PART 9: TRANSPILER BACKEND
// ============================================
class Transpiler {
String emit(Expr expr) {
if (expr is NumberLiteral) return expr.value.toString();
if (expr is BinaryExpr) {
return '(${emit(expr.left)} ${expr.op} ${emit(expr.right)})';
}
throw StateError('Unknown expression');
}
}
// ============================================
// PART 10: DSL WITH METHOD CASCADES
// ============================================
// Dart does not have Groovy-style delegate DSLs
// Method cascades provide a limited embedded DSL
// class Record { ..controlfield('001', 'LIB001'); }
// new Record()..controlfield('001', 'LIB001');The ten parts trace the pipeline from tokens through AST to output. Parts 7 and 8 show the two library-based alternatives to hand-written recursive descent. Part 10 shows the limited embedded-DSL capability that Dart’s method cascades provide.
Quick Reference
Pipeline Stages
| Stage | Input | Output | Dart Approach |
|---|---|---|---|
| Lexer | Source string | Token stream | Hand-written or slex |
| Parser | Token stream | AST | Recursive descent, petitparser, smart_parser |
| Backend | AST | Value or string | Visitor, emitter |
Parsing Approaches
| Approach | Library | Trade-off |
|---|---|---|
| Hand-written recursive descent | None | Full control, more code |
| Parser combinators | petitparser | Composable, type-safe, slightly slower |
| Grammar generator | smart_parser | Grammar in separate file, generated code |
PetitParser Combinators
| Combinator | Meaning |
|---|---|
a & b | Sequence |
| `a | b` |
a.star() | Zero or more |
a.plus() | One or more |
a.optional() | Zero or one |
.map(fn) | Transform result |
.flatten() | Combine to string |
Embedded DSL Limitation
| Feature | Dart Support |
|---|---|
| Method cascades | Yes (..method()) |
| Operator overloading | Yes |
| Delegate binding | No |
| Groovy-style closures | No |
Best Practices
✅ Do This:
// Report errors with position information
throw FormatException('Unexpected token at ${token.position}'); // ✅// Separate lexer, parser, and backend into distinct classes
// Each stage has one responsibility // ✅// Use PetitParser's ExpressionBuilder for operator precedence
final builder = ExpressionBuilder<num>(); // ✅❌ Don’t Do This:
// Combine lexing and parsing into one method
// The error messages become vague and the code becomes untestable // ❌// Use regex to parse nested structures
// Regex cannot handle recursion or balanced delimiters // ❌// Assume Dart supports Groovy-style delegate DSLs
// Method cascades are the only embedded DSL mechanism // ❌Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Parser accepts invalid program | No validation in parser | Check grammar rules |
| Error position is wrong | Token position not tracked | Record position in lexer |
| Infinite loop in parser | Missing advance in repetition | Ensure _advance() in loop |
| Precedence is wrong | Hand-written parser order incorrect | Use ExpressionBuilder |
Real-World Examples
1. Tokenizing an Identifier
tokens.add(Token('IDENT', source.substring(start, _pos), start));2. Parsing a Number
if (token.type == 'NUMBER') return NumberLiteral(num.parse(token.value));3. Binary Expression
left = BinaryExpr(left, op, parseTerm());4. Evaluating a Node
if (expr is BinaryExpr) return evaluate(expr.left) + evaluate(expr.right);5. PetitParser Primitive
builder.primitive(digit().plus().flatten().map(num.parse));6. PetitParser Operator Group
builder.group().left(char('*'), (a, op, b) => a * b);7. Smart Parser Grammar Section
%{ import 'foo.dart'; }%
%%
const SimpleParser();
%%8. Smart Parser Generator
final generator = ParserGenerator(options: options, source: source);
final output = generator.generate();9. Transpiler Emitter
return '(${emit(expr.left)} ${expr.op} ${emit(expr.right)})';10. Method Cascade DSL
new Record()..controlfield('001', 'LIB001')..datafield('100', '');Visual
The Parser Pipeline
┌──────────────────────────────────────────────────────────────┐
│ SOURCE TEXT │
│ "2 + 3 * 4" │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ LEXER │ Character by character │
│ │ │ Produces tokens │
│ └────────┬────────┘ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ TOKENS │ NUMBER(2), TERMINAL(+), NUMBER(3)... │
│ └────────┬────────┘ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ PARSER │ Grammar rules, recursive descent │
│ │ │ or combinators │
│ └────────┬────────┘ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ AST │ BinaryExpr(NumberLiteral(2), │
│ │ │ '+', BinaryExpr(3, '*', 4)) │
│ └────────┬────────┘ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ BACKEND │ Evaluator or code generator │
│ └────────┬────────┘ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ OUTPUT │ 14 (evaluator) or "(2 + (3 * 4))" │
│ └─────────────────┘ │
│ │
│ Each stage has one responsibility. │
│ │
└──────────────────────────────────────────────────────────────┘Parser Approaches
┌──────────────────────────────────────────────────────────────┐
│ HAND-WRITTEN PETITPARSER │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ class Parser { │ │ builder │ │
│ │ Expr parse() │ │ .primitive() │ │
│ │ { ... } │ │ .group() │ │
│ │ } │ │ .left() │ │
│ └─────────────────┘ └─────────────────┘ │
│ │
│ Full control Composable, type-safe │
│ More code Slightly slower │
│ Harder to modify Easy to test │
│ │
│ ┌─────────────────────────────────────────┐ │
│ │ SMART_PARSER │ │
│ │ Grammar file → generated Dart parser │ │
│ └─────────────────────────────────────────┘ │
│ │
│ Grammar separate from code │
│ Generated code is less flexible │
│ │
└──────────────────────────────────────────────────────────────┘Transpiler Backend
┌──────────────────────────────────────────────────────────────┐
│ AST: BinaryExpr(NumberLiteral(2), '+', │
│ BinaryExpr(NumberLiteral(3), '*', NumberLiteral(4)))│
│ │
│ ┌─────────────────┐ │
│ │ EVALUATOR │────► 14 │
│ │ walk tree, │ │
│ │ compute value │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ EMITTER │────► "(2 + (3 * 4))" │
│ │ walk tree, │ │
│ │ build string │ │
│ └─────────────────┘ │
│ │
│ Same AST. Different backend. │
│ Transpiler emits; interpreter evaluates. │
│ │
└──────────────────────────────────────────────────────────────┘Embedded DSL with Cascades
┌──────────────────────────────────────────────────────────────┐
│ DART METHOD CASCADES │
│ │
│ Record() │
│ ..leader = '00000nam' │
│ ..controlfield('001', 'LIB001') │
│ ..datafield('100', 'Author') │
│ │
│ The cascade operator (..) returns the receiver. │
│ Each method call mutates the same object. │
│ │
│ This is the closest Dart comes to an embedded DSL. │
│ No delegate binding, no Groovy-style closures. │
│ │
│ For full DSLs, write a parser. │
│ │
└──────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Lexer | Source string → token stream |
| Parser | Token stream → AST |
| Backend | AST → value or string |
| Recursive descent | Hand-written parser, full control |
| PetitParser | Parser combinators, type-safe, composable |
| Smart Parser | Grammar file → generated Dart parser |
| ExpressionBuilder | Operator precedence handling |
| Transpiler | AST → target-language string |
| Evaluator | AST → computed value |
| Embedded DSL | Method cascades, limited |
Key takeaways:
- The pipeline is lexer, parser, backend. Each stage has one responsibility. The lexer produces tokens with positions. The parser consumes tokens according to a grammar and produces an AST. The backend walks the AST and produces output—a value for an interpreter, a string for a transpiler .
- Hand-written recursive descent is the most direct approach. Each grammar rule becomes a method, and the method calls the methods for the rules it references. It is easy to debug because the call stack mirrors the parse tree. The trade-off is that the code is longer than a combinator grammar .
- PetitParser provides composable, type-safe grammars. The combinators (
&,|,star,plus,map) build parsers from smaller parsers. TheExpressionBuilderhandles operator precedence and associativity, which is the most error-prone part of a hand-written parser . The package includes grammars for JSON, Pascal, Python, and Dart itself, which serve as reference implementations . - Smart Parser separates the grammar from the code. The grammar is written in a file with sections and PEG expressions, and the
ParserGeneratoremits a Dart parser class . This is useful when the grammar changes frequently and the parser code should not be edited by hand. - Dart’s embedded DSL support is limited. Method cascades (
..) provide a chainable object configuration syntax, and operator overloading allows custom operators. But Dart does not have Groovy-style delegate binding or the ability to execute arbitrary host-language code inside a DSL block . For a full DSL, a parser is required.
Remember: Building a transpiler or DSL parser in Dart is a matter of composing three stages, each of which is a plain Dart program. The lexer reads characters and emits tokens. The parser reads tokens and emits a tree. The backend reads the tree and emits output. The libraries in the ecosystem—slex, petitparser, smart_parser—provide the stages or generate them, but they do not change the architecture. The choice between a hand-written parser and a library is a choice between control and convenience. The choice between a transpiler and an interpreter is a choice between emitting a string and returning a value. Both walk the same tree.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!