# Interpreting syntax-parse

This page describes the "interp" implementation of `syntax-parse`.

A pattern is compiled to a "matcher" expression. A matcher is a function that
takes the term to be matched, progress, expectations, and the current match
environment; it produces a backtracking computation producing an updated match
environment.

## Reified environments

The mapping of matched attributes to values, is *reified* as a run-time data
structure, the "runtime environment" or "reified environment" (`renv`). It is a
list of values, with the most recently matched attribute generally stored first,
although it is sometimes reordered for disjunctions (for compatibility) and for
return from syntax classes.

The `renv` stores only the values. The attributes exist only at compile time;
the `aenv` (or attribute environment) is a list of IAttrs that describes the
structure of `renv`. The `aenv` can also contain `#f` to indicate the position
of a wrapped expression bundle (see comments about implementing `~do` below).

- renv - the runtime environment
- aenv - attributes corresponding to slots in renv

### Wrapped expressions

The reified environment must be *reflected* as actual Racket attribute bindings
for the evaluation of any expression that is in the scope of those bindings.

In general, a "wrapped expression" is a lambda expression that receives a `renv`
argument, reflects it as actual attribute bindings, and evaluates the user
expression. Wrapping is done at pattern compilation time based on the `aenv`;
unwrapping is done at run time based on the `renv`.

There is a special case for constant expressions. There is also an indirection
(`bundle-ref`) for expressions in the scope of a `~do` pattern; in that case,
the wrapped expression must close over the `~do` pattern's bindings (if any), so
a closure is created at the end of the `~do` block and pushed onto the `renv`.

## The backtracking monad

Backtracking is handled by the following monad (`BT`):

    ;; BT[t...] = SK[t...] FH FH UndoStack -> Answer
    ;; SK[t...] = FH FH UndoStack t... -> Answer
    ;; FH = (UndoStack Failure -> Answer)

That is, a monadic computation of type `BT[t...]` is a function that takes four arguments:
- a success continuation expecting values of type `t...`
- a failure continuation
- a "cut prompt" (the failure meta-continuation, maybe?)
- an undo stack, actions to be performed upon backtracking (ie, failure)

A success continuation receives the failure continuation, cut prompt, and undo
stack in addition to the result values.

A failure continuation receives the current undo stack and a failure object. It
generally unwinds some prefix of the undo stack before continuing.


## Compilation of patterns

A pattern is compiled to a "matcher", which is a function that takes some
arguments and produces a monadic `BT` computation.

### SinglePattern

A SinglePattern is compiled to a matcher of the type

    Stx Syntax Progress ExpectStack REnv -> BT[REnv]
    
Note: `Syntax` means a syntax object, answers true to `syntax?`; `Stx` means
syntax or syntax-pair or null.

The first argument (`x`) is the term to match; the second (`cx`) is the nearest
enclosing syntax object around `x`. 
For example, suppose we are matching the syntax `S` = `#'(1 2 3 4 5)` against the
pattern `P` = `(x ...)`.
In the first iteration, `x` = `cx` = `S`, but in the next iteration, `x` is a
list of syntax objects while `cx` is still `S`. Thus we avoid repeatedly
allocating new, *artificial* syntax nodes during matching.

The Progress and ExpectStack arguments are used for error reporting.

The REnv argument is used for evaluating ("unwrapping") user expressions. On
success, the matches generated by the pattern are pushed onto the renv.

### HeadPattern

A HeadPattern is compiled to a matcher of the type

    Stx Syntax Progress ExpectStack REnv -> BT[REnv Stx Syntax Progress]

Similar to a SinglePattern matcher, but also returns the tail where to pick up
parsing (represented by the two values `x` and `cx`) and the corresponding
progress at that point.

### ActionPattern

An ActionPattern is compiled to a matcher of the type

    Progress ExpectStack REnv -> BT[REnv]

### Ellipses

Ellipsis matching in its full generality is hairy. The pattern compiler
recognizes several special cases that can be implemented more simply.


## Problem: `#:do` and `~do`

These are difficult, because the `~do` pattern can contain definitions, even
syntax definitions, and those definitions are scoped until the end of the
current *disjunct*. (And sadly there is real Racket code using this feature.)

Implementation strategy:
- From a `~do` pattern, scan forward ("rightward") in the pattern tree
  collecting all *non-trivial* expressions and their associated environments.
  - At run time, the `~do` pattern will push a "bundle" onto the renv.
  - In the (static) attr env, add an entry representing the expression bundle.
- Replace each non-trivial expression with a lookup, using the offset of the
  bundle in the renv and the offset in the bundle of the given expression.
- The `~do` pattern evaluates a block with the user-supplied expr/defs *plus* a
  vector of the gathered (wrapped) expressions. It pushes the resulting vector
  (the "bundle") onto renv.
- If a second `~do` pattern occurs to the right of the first `~do` pattern, then
  the second `~do` pattern's synthesized expression (user-supplied statements
  plus gathered expressions) is treated like any other non-trivial expression
  and gathered by the first pattern.

- Note: When a gathered expression is unwrapped, only the attributes bound since
  the last `~do` pattern need to be bound.

Alternative strategy:
- CPS the patterns themselves.
- CON: adds several more pattern variants, adds another pass

Alternative strategy:
- Fully `local-expand` all expressions (maintaining layered intdefctxs for `~do`
  patterns). That eliminates syntax definitions, and value definitions can be
  reified and reflected in the renv.
- PRO: better discovery of constant exprs, actual attr dependencies
- CON: complicated
