▼Macros and Languages in Racket
►1 Introduction
On this page:
1.1 How to Design Macros
1.2 Designing Your First Macro
1.3 Expansion Contexts and Expansion Order
1.4 Proper Lexical Scoping
1.5 More Implementations of assert
    contents  ← prev  up  next → 

1 Introduction

This section introduces the elements of macro design and illustrates these design elements with simple example macro. It uses the example to introduce some of Racket’s facilities for specifying, implementing, and testing macros.

This guide assumes that you have a basic working knowledge of Racket and functional programming. The Racket Guide is sufficient for the former, and HtDP is good for the latter.

1.1 How to Design Macros

This guide is an attempt to adapt the ideas of How to Design Programs (HtDP) to the design of macros and languages in Racket. The central idea of HtDP is the “design recipe”; the kernel of the design recipe consists of the following four steps:

HtDP instantiates this kernel to teach the foundations of programming. Its specification language is a semiformal language of types including set-based reasoning and parametric polymorphism. Its implementation strategies include structural recursion and case analysis following data type definitions. It instantiates the implementation language to a series of simple Scheme-like functional programming languages, and it provides a testing framework.

Along the way, HtDP fills in the design recipe’s skeleton with idioms, tricks, preferences, and limitations of Scheme-like (and ML-like) mostly-functional programming languages. For example, it demonstrates abstraction via parametric polymorphism and higher-order functions rather than OO patterns. To name some of the limitations: it uses lexical scoping; it avoids reflection (eg, no accessing structure fields by strings); it avoids eval; it treats closures as opaque; it (usually) avoids mutation; and so on. Once you absorb them, these parts of the programming mental model tend to be invisible, until you compare with a language that makes different choices.

This guide instantiates the design recipe kernel as follows: It introduces a specification language called shapes, combining features of grammars, patterns, and types. The implementation strategies are more specialized, but they are still organized around the shapes of macro inputs. The implementation language is Racket with syntax/parse and some other standard syntax libraries.

Along the way, it covers some of the idioms and limitations of the programming model for macros: macros (usually) respect lexical scoping; they must respect the “phase” separation between compile time and run time; they avoid eval; they (usually) treat expressions as opaque; and so on.

1.2 Designing Your First Macro

Suppose we wanted a feature, assert, that takes an expression and evaluates it, raising an error that includes the expression text if it does not evaluate to a true value. The result of the assert expression itself is (void).

Clearly, assert cannot be a function, because a function cannot access the text of its arguments. It must be a macro.

We can specify the shape of assert as follows:
;; (assert Expr) : Expr
That is, the assert macro takes a single argument, an expression, and a use of the assert macro is an expression.

Here are some examples that illustrate the intended behavior of assert:

> (define ls '(1 2 3))
> (assert (> (length ls) 2))
> (assert (even? (length ls)))

assert: assertion failed: (even? (length ls))

In addition to considering the macro’s behavior, it can be useful to consider what code could be used to implement an example use of the macro. The second example, for instance, could be implemented by the following code:
(unless (even? (length ls))
  (error 'assert "assertion failed: (even? (length ls))"))
It would be a bit complicated (although possible) for our assert macro to produce this exact code, because it incorporates the argument expression into a string literal. But there’s no need to produce that string literal at compile time. Here is an equivalent bit of code that produces the same string at run time instead, with the help of quote and error’s built-in formatting capabilities:
(unless (even? (length ls))
  (error 'assert "assertion failed: ~s" (quote (even? (length ls)))))

Lesson: Don’t fixate on the exact code you first write down for the macro’s example expansion. Often, you can change it slightly to make it easier for the macro to produce.

Lesson: It’s often simpler to produce an expression that does a computation at run time than to do the computation at compile time.

That’s our implementation strategy for the assert macro: we will simply use unless, quote, and error. In general, the macro performs the following transformation:
(assert condition)
⇒
(unless condition
  (error 'assert "assertion failed: ~s" (quote condition)))

Before we define the macro, we must import the machinery we’ll use in its implementation:

(require (for-syntax racket/base syntax/parse))

The for-syntax modifier indicates that we need these imports to perform compile-time computation — a macro is implemented by a compile-time function from syntax to syntax. We need racket/base for syntax templates. We need syntax/parse for syntax-parser, which is a pattern-matching utility for syntax objects.

Here is the macro definition:

; (assert Expr) : Expr
(define-syntax assert
  (syntax-parser
    [(_ condition:expr)
     (syntax (unless condition
               (error 'assert "assertion failed: ~e" (quote condition))))]))

Here is an overview of the macro definition:

  • The macro is defined using define-syntax, which takes the macro’s name and a compile-time expression for the macro’s transformer function. By “compile-time expression”, I mean that the expression is evaluated at compile time using the compile-time environment, which is distinct from the normal environment. We initialized the compile-time environment earlier with (require (for-syntax racket/base syntax/parse)).

  • The transformer takes a syntax object representing the macro use and returns a syntax object for the macro’s expansion. This transformer is implemented with syntax-parser, which takes a sequence of clauses consisting of a syntax pattern and a result expression. This macro’s transformer has only one clause.

  • The pattern (_ condition:expr) says that after the macro name (typically represented by the wildcard pattern _) the macro expects one expression, representing a “condition”. The identifier condition is a syntax pattern variable; it is annotated with the syntax class expr. If the clause’s pattern matches the macro use, then its pattern variables are defined and available in syntax templates in the clause’s result expression.

  • The clause’s result expression is a syntax expression, which contains a syntax template. It is similar to quasiquote except that pattern variables do not need explicit unquotes. (It also cooperates with ellipses and some other features; we’ll talk about them later.) When the syntax expression is evaluated, it produces a syntax object with the pattern variables in the template replaced with the terms matched from the macro use. Note that even the occurrence within the quote term gets replaced — pattern variable substitution happens before the quote is interpreted, so a quote in the template is treated like any other identifier.

Finally, we should test the macro. I’ll use rackunit for testing:
> (require rackunit)
Here rackunit is required normally, not for-syntax, because I intend to use it to test the behavior of assert expressions; I don’t intend to test assert’s compile-time transformer function directly.

> (define ls '(1 2 3))
> (check-equal? (assert (> (length ls) 1))
                (void))
> (check-exn exn:fail?
             (lambda ()
               (assert (even? (length ls)))))
What if we want to test uses of assert that might result in compile-time exceptions, like syntax errors? The following does not work:
> (check-exn exn:fail:syntax?
             (lambda ()
               (assert (odd? (length ls)) 'an-extra-argument)))

eval:12:0: assert: unexpected term

  at: (quote an-extra-argument)

  in: (assert (odd? (length ls)) (quote an-extra-argument))

Racket expands and compiles expressions before it evaluates them. The syntax error is detected and raised at compile time (during expansion), but check-exn does not install its exception handler until run time.

One solution is to use eval for this test. This is one of the few “good” uses of eval in Racket programming. Here’s one way to do it:
> (define-namespace-anchor anchor)
> (check-exn exn:fail:syntax?
             (lambda ()
               (parameterize ((current-namespace (namespace-anchor->namespace anchor)))
                 (eval #'(assert (odd? (length ls)) 'an-extra-argument)))))

Another solution is to catch the compile-time exception and “save it” until run time. The syntax/macro-testing library has a form called convert-syntax-error that does that:
> (require syntax/macro-testing)
> (check-exn exn:fail:syntax?
             (lambda ()
               (convert-syntax-error
                (assert (odd? (length ls)) 'an-extra-argument))))

That completes the design of the assert macro. We covered specification, examples, implementation strategy, implementation, and testing.

1.3 Expansion Contexts and Expansion Order

Consider the shape of assert:

;; (assert Expr) : Expr

The first Expr is for the macro’s argument. The second Expr, though, says that assert forms a new kind of expression. But this also points to a limitation of macros: assert is only a new kind of expression.

Not every term in a program matching a macro’s pattern is expanded (that is, rewritten). Macros are expanded only in certain positions, called expansion contexts—essentially, contexts where expressions or definitions may appear. For example, if assert is the macro defined above, then the following occurrences of assert do not count as uses of the macro, and they don’t get expanded:

  • (let ((assert (> 1 2))) 'ok) — This occurrence of assert is in a let-binding; assert is interpreted as a variable name to bind to the value of (> 1 2). In Racket, names like lambda, if, and assert can be shadowed just like variables can!

  • (cond [assert (odd? 4)] [else 'nope]) — This is a syntax error. The cond form treats assert and (odd? 4) as separate expressions, and the use of assert as an expression by itself is a syntax error (the use does not match assert’s pattern).

  • '(assert #f) — This assert occurs as part of a quoted constant.

Note that let and cond are also macros. So we cannot even tell whether a term involving assert is used as an expression until we understand the shapes of the surrounding macros. In particular, the Racket macro expander expands macros in “outermost-first” order, in contrast to nested function calls, which are evaluated “innermost-first.” The outermost-first expansion order is necessary because the macro expander only knows the shapes (and thus the expansion contexts) of primitive syntactic forms; it must expand away the outer macros so that it knows what inner terms need to be expanded.

1.4 Proper Lexical Scoping

Given that assert just expands into uses of unless, error, and so on, perhaps we could interfere with its intended behavior by locally shadowing names it depends on — error, for example. But if we try it, we can see it has no effect:

> (let ([error void])
    (assert (even? (length ls))))

assert: assertion failed: '(even? (length ls))

The assert macro is properly lexically scoped, or hygienic. Roughly, that means that references in assert’s syntax template are resolved in the environment where the macro was defined, not the environment where it is used. This is analogous to the behavior you would get if assert were a function: functions automatically close over their free variables. In the case of macros, it is syntax objects that contain information about the syntax’s lexical context.

In other words, the following “naive” code is the wrong explanation for the expansion of this assert example:
; WRONG
(let ([error void])
  (unless (even? (length ls))
    (error 'assert "assertion failed: ~s" (quote (even? (length ls))))))
Instead, each term introduced by assert carries some lexical context information with it. Here’s a better way to think of the expansion:
(let ([error void])
  (unlessm (even? (length ls))
    (errorm 'assert "assertion failed: ~s" (quotem (even? (length ls))))))
The lexical contexts of error and errorm prevents the use-site local binding of error from capturing the reference errorm.

This example illustrates one half of hygienic macro expansion. We’ll talk about the other half in Proper Lexical Scoping, Part 2.

1.5 More Implementations of assert

Given that we have all of “ordinary” Racket plus several different macro-defining DSLs available for the implementation of assert’s transformer function, there are many other ways we could implement it. This section introduces a few of them.

A (syntax template) expression can be written as #'template instead. That is, #' is a reader macro for syntax. So the assert macro can be defined as follows:
; (assert Expr) : Expr
(define-syntax assert
  (syntax-parser
    [(_ condition:expr)
     #'(unless condition
         (error 'assert "assertion failed: ~e" (quote condition)))]))

The syntax-parser form is basically a combination of lambda and syntax-parse. So the following definition is equivalent:
; (assert Expr) : Expr
(define-syntax assert
  (lambda (stx)
    (syntax-parse stx
      [(_ condition:expr)
       #'(unless condition
           (error 'assert "assertion failed: ~e" (quote condition)))])))
The define-syntax form supports “function definition” syntax like define does, so the following is also allowed:
; (assert Expr) : Expr
(define-syntax (assert stx)
  (syntax-parse stx
    [(_ condition:expr)
     #'(unless condition
         (error 'assert "assertion failed: ~e" (quote condition)))]))

A macro’s transformer function is, in a sense just an ordinary Racket function, except that it exists at compile time. When we imported (for-syntax racket/base) earlier, we made the Racket language available at compile time. We can define the transformer as a separate compile-time function using begin-for-syntax; the definitions it contains are added to the compile-time environment. Then we can simply use a reference to the function as the implementation of assert.
(begin-for-syntax
  ; assert-transformer : Syntax[(_ Expr)] -> Syntax[Expr]
  (define (assert-transformer stx)
    (syntax-parse stx
      [(_ condition:expr)
       #'(unless condition
           (error 'assert "assertion failed: ~e" (quote condition)))])))
; (assert Expr) : Expr
(define-syntax assert assert-transformer)

Note: There are two differences between assert and assert-transformer. The name assert is bound as a macro in the normal environment (also called the run-time environment or the phase-0 environment), whereas the name assert-transformer is bound as a variable in the compile-time environment (also called the transformer environment or the phase-1 environment). Both of them are associated with a compile-time value, but assert-transformer is not a macro; if you replace assert with assert-transformer in the tests above, they will not even compile. Likewise, you cannot use assert in a compile-time expression, either as a macro or as a variable. The separation of run-time and compile-time environments is part of Racket’s phase separation.

In addition to syntax/parse, Racket also inherits Scheme’s older macro-definition DSLs: syntax-rules and syntax-case, and they are used in much existing Racket code. Here are versions of assert written using those systems:

(define-syntax-rule (assert condition)
  (unless condition
    (error 'assert "assertion failed: ~s" (quote condition))))
 
(define-syntax assert
  (syntax-rules ()
    [(_ condition)
     (unless condition
       (error 'assert "assertion failed: ~s" (quote condition)))]))
 
(define-syntax (assert stx)
  (syntax-case stx ()
    [(_ condition)
     #'(unless condition
         (error 'assert "assertion failed: ~s" (quote condition)))]))

For a macro as simple as assert, there isn’t much difference. All of the systems share broadly similar concepts such as syntax patterns and templates. The syntax/parse system evolved out of syntax-case; syntax/parse adds a more sophisticated pattern language and a more expressive way of organizing compile-time syntax validation and computation.

All of these pattern-matching DSLs are simply aids to writing macros; they aren’t necessary. It’s possible to write the macro by directly using the syntax object API. Here’s one version:

(define-syntax assert
  (lambda (stx)
    (define parts (syntax->list stx)) ; parts : (U (Listof Syntax) #f)
    (unless (and (list? parts) (= (length parts) 2))
      (raise-syntax-error #f "bad syntax" stx))
    (define condition-stx (cadr parts)) ; condition-stx : Syntax[Expr]
    (define code
      (list (quote-syntax unless) condition-stx
            (list (quote-syntax error) (quote-syntax 'assert)
                  (quote-syntax "assertion failed: ~s")
                  (list (quote-syntax quote) condition-stx))))
    (datum->syntax (quote-syntax here) code)))

Briefly, syntax->list unwraps a syntax object one level and normalizes it to a list, if possible (the terms (a b c) and (a . (b c)), while both “syntax lists”, have different syntax object representations). It is built on top of the primitive operation syntax-e. The quote-syntax form is the primitive that creates a syntax object constant for a term that captures the lexical context of the term itself. The lexical context can be transferred to a tree using datum->syntax; it wraps pairs, atoms, etc, but it leaves existing syntax objects unchanged.

Here is a variant of the previous definition that uses quasisyntax (reader abbreviation #`) and unsyntax (reader abbreviation #,) to construct the macro’s result:

(define-syntax assert
  (lambda (stx)
    (define parts (syntax->list stx)) ; parts : (U (Listof Syntax) #f)
    (unless (and (list? parts) (= (length parts) 2))
      (raise-syntax-error #f "bad syntax" stx))
    (define condition-stx (cadr parts)) ; condition-stx : Syntax[Expr]
    #`(unless #,condition-stx
        (error 'assert "assertion failed: ~s" '#,condition-stx))))

It is not a goal of this guide to introduce you to every bit of machinery that can be used to implement macros. In general, this guide will stick to the syntax/parse system for macro definitions, and it uses the #' abbreviation for syntax expressions. It will sometimes be necessary to use the lower-level APIs (such as syntax->list, #` and #,) to perform auxiliary computations.

On the other hand, it is a goal of this guide to discuss and compare different implementation strategies. So the following sections do often present multiple implementations of the same macro according to different strategies.

    contents  ← prev  up  next →