The Xmacro Library

This library provides functionality to implement XMacros and code templating.

This is done by defining lists of expansions which will later be substituted at given places. Xmacros support their own scopes that can be nested. This allows for fine control of what becomes expanded.

The syntax is simple and powerful. It is especially useful for generating repetitive code, such as trait implementations, where the same code pattern needs to be repeated with different types in a similar fashion. Keeping tables of identifiers, data, documentation and so on in sync when generating code.

The xmacro expansions are invoked with xmacro_expand(input: TokenStream) and xmacro_expand_items(input: TokenStream).

Initial Examples

Simple Expansion

The most basic form are named or unnamed definitions that are straightforward substituted each-by-each (see later about this).

The following three examples will all expand to:

# xmacro_lib::doctest_ignore!{
impl Trait<Foo> for MyType<Foo> {}
impl Trait<Bar> for MyType<Bar> {}
impl Trait<Baz> for MyType<Baz> {}
# }

Note that each of these examples is defined in the top-level scope. In a real use case you probably want to put xmacros in scopes ${...} to limit the expansion to the code in concern.

Unnamed definitions:

An unnamed definition of a list of parenthesized expansions within $(...) will expand in place. Unnamed definitions can be referenced by the position of their appearance within the same local scope. Here we have only one definition, $0 references the first and only one.

# xmacro_lib::CHECK_EXPAND!({
impl Trait<$((Foo)(Bar)(Baz))> for MyType<$0> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# })

Or even simpler, when the substitutions are single tokens then the parenthesis around can be omitted:

# xmacro_lib::CHECK_EXPAND!({
impl Trait<$(Foo Bar Baz)> for MyType<$0> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# })

Named definition:

A named definition $(name: ...) will not expand in place, it can be used later by a substitution. The substitution can be by name or position.

# xmacro_lib::CHECK_EXPAND!({
// Definition as T which also is the first (position 0) definition
$(T: (Foo)(Bar)(Baz))
// Then use it either by $T or $0
impl Trait<$T> for MyType<$0> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# });

Named, in place definition:

A named definition with a dollar sign in front of the name $($name: ...) will expand in place and can be later used by name or index as well. This is often sufficient for simple one-line cases.

# xmacro_lib::CHECK_EXPAND!({
// Definition and substitution in place.
impl Trait<$($T: (Foo)(Bar)(Baz))> for MyType<$T> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# })

Table Definition

It is possible to give a header line with multiple names followed by interleaved expansions. Such table are then expanded for each row.

For example one can create a enum and a table to related names in tandem by:

# xmacro_lib::CHECK_EXPAND!({
// Define table data, the first line contains the headers with names followed by colons
// followed by rows defining the data. The scope becomes expanded per row. We can omit the
// parenthesis around data items here since they are single tokens.
$(
    // the headers
    id:  text:
    // the data
    Foo  "foo"
    Bar  "bar"
    Baz  "baz"
)

// enum where each variant relates to an integer index in the following array
#[repr(usize)]
enum MyEnum {
    // Use a child scope, which expands '$id' with a comma appended.
    // Note how $id is automatically imported from the outer scope.
    ${$id,}
}

// A static array of texts, `$#text` expands to the number of elements in `text`
static MY_TEXTS: [&'static str; $#text]  = [
    // expand each of `$text` with a comma appended
    ${$text,}
];
# } == {

// expands to:

#[repr(usize)]
enum MyEnum {
    Foo,
    Bar,
    Baz,
}

static MY_TEXTS: [&'static str; 3]  = [
    "foo",
    "bar",
    "baz",
];
# });

Syntax

Xmacros can substitute almost everything. Things must tokenize into a TokenStream, the only requirement is that literals must be properly formatted and all opening brackets must be closed.

Overview

All syntactic elements relevant to the xmacro syntax start with the dollar sign $. This syntax contains the elements described in detail below:

• Scopes for xmacro definitions using curly braces:

  Scopes are the base on which xmacro expansion happens. They allow for nested definitions and substitutions. A scope defines how entities inside become expanded. There is an implicit top level scope, thus many examples here omit defining a scope.

  # xmacro_lib::doctest_ignore!{
  ${
      ...
  }
  # }
  

• Definitions using parentheses:

  • A Definition is local to its scope.
  • It defines a non empty list expansions which are iterated when expanding the scope.
  • Definitions that are not used by substitution in a scope won't be part of the expansion processs
  • Definitions are expanded each by each with each other.
  • Table definitions will expand by rows.

  # xmacro_lib::doctest_ignore!{
  $( ... )   // definition
  # }
  

• Substitutions, either named or positional:

  Substitution are the way to refer to previously defined expansion list by name or position.

  # xmacro_lib::doctest_ignore!{
  $0    // positional substitution
  $foo  // named substitution
  # }
  

• Directives and special expansions:

  # xmacro_lib::doctest_ignore!{
  $?0             // the current index within the expansion list
  $#foo           // the length of the expansion list of a definition
  $+prefix$suffix // identifier concatenation
  $$              // escapes a literal dollar character
  # }
  

Detailed Syntax

Definitions

A Xmacro definition can be named or unnamed, hidden or in-place.

In either way a definition holds a list of expansions (code fragments) to expand to. Usually these are written in parenthesis, in case these expansions are single tokens the parenthesis can be omitted, note the braced and bracketed groups count as single token and preserve the outer brace or bracket. When one wants expand parenthesis these needs to be double wrapped in parenthesis.

Unnamed definitions can only be referenced by their numeric position in the order their definitions appearance. They are local to their scope.

Named definitions use an identifier followed by a colon which can be used to reference the definition. Unlike unnamed definitions, named definitions can be looked up from child scopes to import a definition for a parent scope.

Hidden definitions will not be substitute at the place of their definition but can be referenced later. This is useful for putting all definitions at the begin of a scope and using them later, possibly importing them in nested scopes. In-place definitions will be substituted at the place of their definition and can be referenced later as well.

This leads to following forms (unnamed-hidden is not supported).

• $((a)(b)) - unnamed, in-place:
  Defining an unnamed definition will implicitly substitute it in place. Can be referenced by a positional number. This is useful for defining expansions that are used only once or used in one-liners.
• $(name: (a)(b)) - named, hidden:
  Using a named definition allows it to be referenced by its name. It is not substituted at the place of its definition, this is useful when one wants to put all definitions at the begin of a scope. Named definitions are required when child scopes want to import definitions from their parents.
• $($name: (a)(b)) - named, in-place:
  Defining a named expansion with a $ in front of the name marks it to be substituted in place. This is a efficient way to define something within a fragment of code and substitute it later again.
• $(name0: nameN: (a0) (aN) (b0) (bN)) - table:
  A definition with multiple names constitutes a table, the expansions defined in a table will be expanded by rows and not each by each like unamed or single named definitions. This syntax allows writing data in vertical tables (not in single line as shown here). See example below.

The position of the xmacro definitions for each scope starts with zero. Using a named substitution to import a definition from a parent scope will reserve the next position for it.

For the name: part there is a name@matching: syntax for reserved for future extension. Currently the @matching part is ignored.

Note:
xmacro expansions can only contain Rust tokens, no recursive xmacro definitions or substitutions. This is intentionally chosen for simplicity for now.

Vertical Table Example

# xmacro_lib::doctest_ignore!{
// Defines a table
$(
    lifetime: T:         param:           push:
    ()        (char)     (c: char)        (push(c))
    (<'a>)    (&'a char) (c: &char)       (push(*c))
    (<'a>)    (&'a str)  (s: &str)        (push_str(s))
    ()        (CowStr)   (cowstr: CowStr) (push_str(cowstr.as_str()))
    ()        (SubStr)   (substr: SubStr) (push_str(&substr))
)

// Expand this code for each row in the table
impl $lifetime Extend<$T> for CowStr {
    fn extend<I: IntoIterator<Item = $T>>(&mut self, iter: I) {
        iter.for_each(move |$param| self.$push);
    }
}
# }

Substitutions

Substitutions are used to reference an earlier defined expansion lists. They are written as dollar-sign followed by the position or the name of a named xmacro definition. Positional substitutions are only valid for local scope, indexing starts at zero. A named substitution which refers to a name from a parent scope will import that definition into the current scope on first use. This will also reserve a position for it.

# xmacro_lib::CHECK_EXPAND!({
// outer scope
$(foo: this_is_foo)
$foo
${
    // inner scope
    $(zero) $(one) $foo becomes $2
    $0 $1 $2
}
# } == {

// expands to

this_is_foo
zero one this_is_foo becomes this_is_foo
zero one this_is_foo
# });

Unresolved substitutions will lead to a compile error. Everything has to be defined before being used.

Scopes

Scopes cover the code where expansion is done. They are evaluated inside-out and expanding all substitutions. There is always a implicit top-level scope, expansion of this level scope can have special semantics (see [xmacro_expand_items()]).

Expansion in a scope only happens for substitutions that are actually used. Using a substitution by name that is not defined in the current scope tries to import this from the parent scopes. If that fails a error is returned.

Each-by-Each Expansion

Expansion is the product of each used substitution with each other. This is used for unnamed or non table definitions.

# xmacro_lib::CHECK_EXPAND!({
$(name:   (one) (two))
$(number: (1)   (2))
$(roman:  (I)   (II))
$name:$number:$roman;
# } == {

// expands to:

one:1:I;
one:1:II;
one:2:I;
one:2:II;
two:1:I;
two:1:II;
two:2:I;
two:2:II;
# });

Row expanded Expansion

Using a table definition will expand by rows.

# xmacro_lib::CHECK_EXPAND!({
$(
    name:   number: roman:
    one     1       I
    two     2       II
)
$name:$number:$roman;
# } == {

//expands to:

one:1:I;
two:2:II;
# });

Note how this can be mixed:

# xmacro_lib::CHECK_EXPAND!({
$(name:   (one) (two))
$(
    number: roman:
    1       I
    2       II
)
$name:$number:$roman;
# } == {

// expands to:

one : 1 : I ;
one : 2 : II ;
two : 1 : I ;
two : 2 : II ;
# });

Special Expansions

$?definition expands to the current index in the expansion-list of definition. This marks definition to be used in the scope.

$#definition expands to the number of defined expansions of definition. This does not mark definition to be used in the scope.

# xmacro_lib::CHECK_EXPAND!({
$((a)(b)(c)) $?0 $#0
# } == {

// expands to:

a 0 3
b 1 3
c 2 3
# });

$+prefix$suffix identifier constructor. Creates a new idenifier by concatenating prefix and $suffix. The prefix must be a identifier $suffix must be a definition that expands to strings that are valid identifiers.

# xmacro_lib::CHECK_EXPAND!({
$(suffix: a b c 1 2 3 foobar)
$+test_$suffix
# } == {

// expands to:

test_a
test_b
test_c
test_1
test_2
test_3
test_foobar
# });

Escaping the $ character

The $ character is used to start xmacro syntax elements. To use a literal $ character, double it.

Expansion Semantic

• Everything has to be defined before used.
  When using $substitution either by name or position it has to be already defined. For named substitutions this can be defined in a parent scopes which will import the definition into the current scope.

• Redefinition is an error.
  Trying to redefine the same name in a scope again will return an error.

• Importing creates takes a position.
  When a definition from a parent scope is used by name in a child scope, it actually becomes imported to the local scope, thus it also gets the next position assigned.

• No empty definitions.
  The semantics of empty definitions are not clear yet,

  • Should they suppress output (formally correct but surprising)
  • Or act like a single empty definition $(()), likely what one expects.

  For the time being we just disallow empty definitions.

• Expand only whats used.
  Only definitions that are referenced by name or position (or in-place) will be part of the xmacro substitution. Things that are not used wont create spurious substitutions.

• When not definitions are referenced, then code stays verbatim.
  This allows defining global things in the top level scope while using them only in child scopes.

• The index-of $?definition will cause expansion.
  Even when $definition is not used, it will mark it part if the expansion loop and expand each-by-each or per-row.

• The length-of $#definition will not cause expansion.
  This is like a constant and is not part if each-by-each or per row expansion.

Tips & Tricks

• Code templating

  When you use only one expansion per definition then xmacro acts like a normal code template engine.

  # xmacro_lib::CHECK_EXPAND!({
  $(name: MyStruct)
  $(type: String)
  
  struct $name {
      value: $type
  }
  
  impl $name {
      pub fn new(val: $type) -> $name {
          $name { value: val }
      }
  
      pub fn value(&self) -> &$type {
          &self.value
      }
  }
  # } == {
  # struct MyStruct {
  #   value: String
  # }
  # impl MyStruct {
  #   pub fn new(val: String) -> MyStruct {
  #         MyStruct { value: val }
  #     }
  #     pub fn value(&self) -> &String {
  #         &self.value
  #     }
  # }
  # });
  

• Repeated expansion

  When you want to repeat something for a number of times then empty definitions that are each-by-each expanded can be used to to multiply the content of a scope:

  # xmacro_lib::CHECK_EXPAND!({
  ${
      $($TEN: ()()()()()()()()()())
      $($THREE: ()()())
      // Note that named definitions are only used for clarity here,
      // `$(()()()()()()()()()()) $(()()())` would work as well.
      // Will repeat `this` TEN*THREE == 30 times
      this
  }
  # } == {
  #     this this this
  #     this this this
  #     this this this
  #     this this this
  #     this this this
  #     this this this
  #     this this this
  #     this this this
  #     this this this
  #     this this this
  # });