From characters to located tokens

A parser that stares at an undifferentiated character stream spends its life guessing. Where does the identifier end? Is = part of == or a lonely assignment? Did a string already open?

Tokenization (lexical analysis) cuts the stream into tokens: small classified chunks with spelling and source locations. Later stages stop arguing about character boundaries and start arguing about meaning.

Token boundaries and spelling

A token carries at least:

a kind (identifier, string literal, keyword-like form, delimiter, operator, ...)
the authored spelling as it appeared in source
a source location span pointing back at the file

Take the page import:


    import @greeting { greet }

A lexer might emit a sequence in this spirit:

Spelling	Kind family
`import`	keyword or identifier reserved by grammar
`@greeting`	package or module path atom
`{`	delimiter
`greet`	identifier
`}`	delimiter

Exact kind names live in the tokenizer. The teaching point stays fixed: boundaries enter here, meaning does not.

Identifiers, delimiters, operators and literals each become discrete pieces. The parser will arrange them. The type checker will judge them. The lexer only marks the boundaries.

Diagram placeholder: Source characters become located token kinds

Caption: simplified lexer view. Question: where do boundaries enter the pipeline?

Source locations

Every useful diagnostic eventually points at text the author wrote. Locations make that possible without forcing every later stage to keep the entire file string beside every node forever.

A span says "these characters in this source identity." When interface binding later fails to find greet, the diagnostic can underline the import name rather than shrug at "somewhere in the module."

Tokenization attaches locations. It does not decide that greet is a function. That decision waits for binding and AST.

Diagram placeholder: Token spans point back to exact source ranges

Caption: exact diagnostic principle, simplified data shape. Question: how can later stages point to authored text without reparsing?

Code and document source kinds

Not every file starts in the same lexical mode.

Ordinary .moth starts in code mode.
Moth template .mtf starts in an implicit template body while preserving original source locations.
Plain .md is prepared before tokenization and does not use a tokenizer entry mode in the same way.

Those paths converge. Moth template preparation synthesises an ordinary private content #String declaration whose initializer is a Markdown template over the body tokens. Markdown preparation can contribute the same declaration shape with a synthetic string literal. Later stages see ordinary declarations, not a forever-special document dialect in HIR.

The greeting project uses .moth files. Document kinds still matter because the same compiler owners serve docs sites and content imports without a second semantic pipeline.

Diagram placeholder: moth, mtf and md use different entry preparation then converge

Caption: teaching simplification. Question: why avoid duplicate pipelines for documents and code?

Lexical failures

Some mistakes die at the first cut:

invalid characters or escapes
malformed delimiters
symbolic spacing rules for operators, assignments and mutable declarations
numeric text that fails shared lexical grammar

numeric_text owns shared numeric grammar, normalisation, separators and exponents. Lexical numeric failure is not yet a semantic Number versus Byte debate. Semantic numeric types arrive later. Keep those layers separate when you read diagnostics.

Boundary of ownership

Tokenization recognises spelling, locations and delimiter context. It may reject unsupported directive forms at the syntax level through the merged directive registry.

It does not:

resolve imports
build types
fold constants
understand templates as semantic structures
decide target support

If a stage after the lexer rediscovers token boundaries from raw text, something has slipped.

What can fail here

invalid character, escape, delimiter or lexical spacing
source-kind preparation failure
import, type and template semantic errors that only look lexical when misdiagnosed

Defer semantic errors to their owning stages. A good lexer stays humble.

Why Moth does it this way

Alternative: give each source kind a separate parser and backend pipeline.

Choice: adapt source kinds early, then share declaration and semantic stages.

Benefit: one semantic owner for ordinary declarations and runtime lowering. Document content becomes data the rest of the compiler already understands.

Cost: source-kind adapters must preserve locations and keep boundaries crisp.

Roadmap and current status

Accepted design: located tokens and retained source preparation.
Current implementation: progress rows for source-kind coverage.
Queued work: numeric roadmap may change later semantic ownership, not lexical token boundaries.
Deferred or outside scope: tokenization does not infer target support or type identity.

Exit state

You leave with located tokens and source-kind prepared payloads. Next: which declarations and import relationships those tokens describe.