The Xmacro Library

This library provides functionality to implement XMacros.

These Xmacros can be nested using scopes. Within each of these scopes, one can bind a list of expansions to a definition. When scopes are expanded the values of these definitions are substituted either on a each-by-each base or by iterating over all rows in a table.

The syntax is simple and powerful. It is especially useful for generating repetitive code, such as trait implementations, where the same code pattern needs to be repeated with different types in a similar fashion. Keeping tables of identifiers, data, documentation and so on in sync when generating code.

The xmacro expansions are invoked with xmacro_expand(input: TokenStream) and xmacro_expand_items(input: TokenStream).

Initial Examples

Simple Expansion

The most basic form are named or unnamed definitions that are straightforward substituted each-by-each (see later about this).

The following three examples will all expand to:

# xmacro_lib::doctest_ignore!{
impl Trait<Foo> for MyType<Foo> {}
impl Trait<Bar> for MyType<Bar> {}
impl Trait<Baz> for MyType<Baz> {}
# }

Note that each of these examples is defined in the top-level scope. In a real use case you probably want to put xmacros in scopes ${...} to limit the expansion to the code in concern.

Unnamed definitions:

An unnamed definition of a list of parenthesized expansions within $(...) will expand in place. Unnamed definitions can be referenced by the position of their appearance within the same local scope. Here we have only one definition, $0 references the first and only one.

# xmacro_lib::CHECK_EXPAND!({
impl Trait<$((Foo)(Bar)(Baz))> for MyType<$0> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# })

Or even simpler, when the substitutions are single tokens then the parenthesis around can be omitted:

# xmacro_lib::CHECK_EXPAND!({
impl Trait<$(Foo Bar Baz)> for MyType<$0> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# })

Named definition:

A named definition $(name: ...) will not expand in place, it should be later used in a substitution. The substitution can be by name or position.

# xmacro_lib::CHECK_EXPAND!({
// Definition as T which also is the first (position 0) definition
$(T: (Foo)(Bar)(Baz))
// Then use it either by $T or $0
impl Trait<$T> for MyType<$0> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# });
# xmacro_lib::CHECK_EXPAND!({
# $(T: Foo Bar Baz)
# impl Trait<$T> for MyType<$0> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# });

Named, in place definition:

A named definition with a dollar sign in front of the name $($name: ...) will expand in place. and can be later used by name. This is often sufficient for simple one-line cases.

# xmacro_lib::CHECK_EXPAND!({
// Definition and substitution in place.
impl Trait<$($T: (Foo)(Bar)(Baz))> for MyType<$T> {}
# } == {
# impl Trait<Foo> for MyType<Foo> {}
# impl Trait<Bar> for MyType<Bar> {}
# impl Trait<Baz> for MyType<Baz> {}
# })

Table Definition

Aside from normal definitions which are each-by-each expanded, xmacro has a syntax to define vertical tables that are expanded for each row.

Table definitions are a dollar sign with the table followed in square brackets.

For example one can create a enum and a table to related names in tandem by:

# xmacro_lib::CHECK_EXPAND!({
// Define table data, the first line contains the headers with names followed by colons
// followed by rows defining the data. The scope becomes expanded per row. We can omit the
// parenthesis around data items here since they are single tokens.
$[
    // the headers
    id:  text:
    // the data
    Foo  "foo"
    Bar  "bar"
    Baz  "baz"
]

// enum where each variant relates to an integer index in the following array
#[repr(usize)]
enum MyEnum {
    // Use a child scope, which expands '$id' from the outer scope
    // with a comma appended.
    ${$id,}
}

// A static array of texts, `$#text` expands to the number of rows in `text`
static MY_TEXTS: [&'static str; $#text]  = [
    // expand each of `$text` with a comma appended
    ${$text,}
];
# } == {

// expands to:

#[repr(usize)]
enum MyEnum {
    Foo,
    Bar,
    Baz,
}

static MY_TEXTS: [&'static str; 3]  = [
    "foo",
    "bar",
    "baz",
];
# });

Syntax

Xmacros can substitute and expand almost everything. Things must tokenize into a TokenStream, this means literals must be properly formatted, and all opening brackets must be closed.

Overview

All syntactic elements relevant to the xmacro syntax start with the dollar sign $. This syntax contains the elements described in detail below:

• Scopes for xmacro definitions using curly braces:

  Scopes are the base on which xmacro expansion happens. They allow for nested definitions and substitutions. A scope defines how entities inside become expanded. There is an implicit top level scope.

  # xmacro_lib::doctest_ignore!{
  ${
      ...
  }
  # }
  

• Definitions using parentheses or square brackets:

  • A Definition is local to its scope.
  • It defines a non empty list expansions which are iterated when expanding the scope.
  • A scope defines the mode in which definitions are expanded.
    • Single definitions are expanded each-by-each.
    • Table definitions will expand linear.
    • This can be overridden by special directives.

  # xmacro_lib::doctest_ignore!{
  $( ... )   // single definition
  $[ ... ]   // table definition
  # }
  

• Substitutions, either named or positional:

  Substitution are the way to refer to previously defined expansion list by name or position.

  # xmacro_lib::doctest_ignore!{
  $0    // positional substitution
  $foo  // named substitution
  # }
  

• Directives and special expansions:

  # xmacro_lib::doctest_ignore!{
  $?0     // the current index within the expansion list
  $#foo   // the length of the expansion list of a definition
  $:flag  // directive that alters a flag
  $$      // escapes a literal dollar character
  # }
  

Detailed Syntax

Definitions

A Xmacro definition can be named or unnamed, hidden or in-place.

In either way a definition holds a list of expansions (code fragments) to expand to. Usually these are written in parenthesis, in case these expansions are single tokens the parenthesis can be omitted, note the braced and bracketed groups count as single token and preserve the outer brace or bracket. When one wants expand parenthesis these needs to be wrapped parenthesis.

Unnamed definitions can only be referenced by their numeric position in the order their definitions appearance. They are local to their scope.

Named definitions use an identifier followed by a colon which can be used to reference the definition. Unlike unnamed definitions, named definitions can be looked up from child scopes to import a definition for a parent scope.

Hidden definitions will not be substitute at the place of their definition but can be referenced later. This is useful for putting all definitions at the begin of a scope and using them later. In-place definitions will be substituted at the place of their definition and can be referenced later as well.

This leads to following three forms (unnamed-hidden is not supported).

• $((a)(b)) - unnamed, in-place:
  Defining an unnamed definition will implicitly substitute it in place. Can be referenced by a positional number. This is useful for defining expansions that are used only once or used in one-liners.
• $(name: (a)(b)) - named, hidden:
  Using a named definition allows it to be referenced by its name. It is not substituted at the place of its definition, this is useful when one wants to put all definitions at the begin of a scope. Named definitions are required when child scopes want to import definitions from their parents.
• $($name: (a)(b)) - named, in-place:
  Defining a named expansion with a $ in front of the name marks it to be substituted in place. This is a efficient way to define something within a fragment of code and substitute it later again.

The position of the xmacro definitions for each scope starts with zero. Using a named substitution to import a definition from a parent scope will reserve the next position for it.

Note:
xmacro expansions can only contain Rust tokens, no recursive xmacro definitions or substitutions. This is intentionally chosen for simplicity for now.

Table Definitions

Table definitions offer a vertical table syntax which is often easier to maintain than single definitions. They use square brackets instead parenthesis. Tables become expanded for each row instead each-by-each. The table won't expand in place (hidden), it should precede the code using it.

• $[name0: nameN: (a0) (aN) (b0) (bN)] - vertical table form
  This turns the scope into a linear expanded scope and allows to list all items in a vertical table (not in single line as shown here). See example below. When mixed with other definitions all definitions have to have the same number of expansions.

# xmacro_lib::doctest_ignore!{
// Define a table
$[
    lifetime: T:         param:           push:
    ()        (char)     (c: char)        (push(c))
    (<'a>)    (&'a char) (c: &char)       (push(*c))
    (<'a>)    (&'a str)  (s: &str)        (push_str(s))
    ()        (CowStr)   (cowstr: CowStr) (push_str(cowstr.as_str()))
    ()        (SubStr)   (substr: SubStr) (push_str(&substr))
]

// Expand this code for each row in the table
impl $lifetime Extend<$T> for CowStr {
    fn extend<I: IntoIterator<Item = $T>>(&mut self, iter: I) {
        iter.for_each(move |$param| self.$push);
    }

    fn extend_one(&mut self, $param) {
        self.$push;
    }
}
# }

Substitutions

Substitutions are used to reference an earlier defined expansion lists. They are written as dollar-sign followed by the position or the name of a named xmacro definition. Positional substitutions are only valid for local scope, indexing starts at zero. A named substitution which refers to a name from a parent scope will import that definition into the current scope on first use. This will also reserve a position for it.

# xmacro_lib::CHECK_EXPAND!({
// outer scope
$(foo: this_is_foo)
$foo
${
    // inner scope
    $(zero) $(one) $foo becomes $2
    $0 $1 $2
}
# } == {

// expands to

this_is_foo
zero one this_is_foo becomes this_is_foo
zero one this_is_foo
# });

Unresolved substitutions will lead to a compile error. Everything has to be defined before being used.

Scopes

Scopes encapsulate and define the way how expansion is done, either each-by-each or by rows. They are evaluated inside-out and expanding all substitutions. There is always a implicit top-level scope, expansion of this level scope can have special semantics.

Expansion in a scope only happens for definitions that are actually used.

Each-by-Each Scopes

This is the default for scopes. Expansion is the product of each used substitution with each other.

# xmacro_lib::CHECK_EXPAND!({
$(name:   (one) (two))
$(number: (1)   (2))
$(roman:  (I)   (II))
$name:$number:$roman;
# } == {

// expands to:

one:1:I;
one:1:II;
one:2:I;
one:2:II;
two:1:I;
two:1:II;
two:2:I;
two:2:II;
# });

Row expanded Scopes

Using a table definition will turn a scope into rows expansion mode. All used definitions in a rows expanded scope must have the same number of expansions in their list. This is enforced when the table definition syntax is used but one need to be careful when table and normal definitions are mixed.

# xmacro_lib::CHECK_EXPAND!({
$[
    name:   number: roman:
    one     1       I
    two     2       II
]
$name:$number:$roman;
# } == {

//expands to:

one:1:I;
two:2:II;
# });

Special Expansions

$?definition expands to the current index in the expansion-list of definition. This marks definition to be used in the scope.

$#definition expands to the number of defined expansions of definition. This does not mark definition to be used in the scope.

# xmacro_lib::CHECK_EXPAND!({
$((a)(b)(c)) $?0 $#0
# } == {

// expands to:

a 0 3
b 1 3
c 2 3
# });

Directives

$:flag explicitly modifies a flag. Directives apply instantly, later actions may override the directive action.

Following flags are defined:

• $:eachbyeach
  This scope becomes each-by-each expanded.
• $:rows
  This scope becomes 'linear' expanded.

Escaping the $ character

The $ character is used to start xmacro syntax elements. To use a literal $ character, double it.

Expansion Semantic

• Everything has to be defined before used.
  When using $substitution either by name or position it has to be already defined. For named substitutions this can be defined in a parent scopes which will import the definition into the current scope.
• Redefinition is an errors.
  Trying to redefine the same name in a scope again will return an error.
• Importing creates takes a position.
  When a definition from a parent scope is used by name in a child scope, it actually becomes imported to the local scope, thus it also gets the next position assigned.
• No empty definitions.
  The semantics of empty definitions are not clear yet,
  • Should they suppress output (formally correct but surprising)
  • Or act like a single empty definition $(()), likely what one expects. For the time being we just disallow empty definitions.
• Expand only whats used.
  Only definitions that are referenced by name or position (or in-place) will be part of the xmacro substitution. Things that are not used wont create spurious substitutions.
• When nothing definitions are referenced, then code stays verbatim.
  This allows defining global things in the top level scope while using them only in child scopes.
• The index-of $?definition will cause expansion.
  Even when $definition is not used, it will mark it part if the expansion loop and expand each-by-each or per-row.
• The length-of $#definition will not cause expansion.
  This is like a constant and is not part if each-by-each or per row expansion.