6 releases
new 0.2.2 | Mar 24, 2023 |
---|---|
0.2.1 | Mar 24, 2023 |
0.1.7 | Mar 19, 2023 |
#274 in Procedural macros
128 downloads per month
Used in 5 crates
(2 directly)
26KB
526 lines
Macro Magic 🪄
This crate provides two powerful proc macros, #[export_tokens]
and import_tokens!
. When
used in tandem, these two macros allow you to mark items in other files (and even in other
crates, as long as you can modify the source code) for export. The tokens of these items can
then be imported by the import_tokens!
macro using the path to an item you have exported.
Two advanced macros, import_tokens_indirect!
and read_namespace!
are also provided when the
"indirect" feature is enabled. These macro are capable of going across crate boundaries without
complicating your dependencies and can return collections of tokens based on a shared common
prefix.
Among other things, the patterns introduced by Macro Magic can be used to implement safe and
efficient coordination and communication between macro invocations in the same file, and even
across different files and different crates. This crate officially supercedes my previous
effort at achieving this, macro_state, which was
designed to allow for building up and making use of state information across multiple macro
invocations. All of the things you can do with macro_state
you can also achieve with this
crate, albeit with slightly different patterns.
macro_magic
is designed to work with stable Rust.
Example
Let's say you have some module that defines a bunch of type aliases like this:
// src/bar/baz.rs
pub mod foo {
type Foo = u32;
type Bar = usize;
type Fizz = String;
type Buzz = bool;
}
And let's say you are writing some proc macro somewhere else, and you realize you really need
to know what types have and have not been defined in the bar::baz::foo
module shown above,
perhaps so you can provide default values for these type aliases if they are not present.
#[proc_macro]
pub fn my_macro(tokens: TokenStream) -> TokenStream {
// ...
let foo_tokens: TokenStream2 = ???
// ...
}
We need the tokens from some item that hasn't been passed to our macro here. How can we get them?
Well, you can attach the #[export_tokens]
attribute macro to the foo
module as follows:
// src/bar/baz.rs
use macro_magic::export_tokens;
#[export_tokens]
pub mod foo {
type Foo = u32;
// ...
}
Now you can import the tokens for the entire foo
module inside of my_macro
even though
they are in different crates. The only caveat is that you have to import the foo
module to
the context where you are writing your macro, like so:
use bar::baz::foo;
use macro_magic::import_tokens;
#[proc_macro]
pub fn my_macro(tokens: TokenStream) -> TokenStream {
let foo_tokens: TokenStream = import_tokens!(foo).into(); // type is TokenStream2
let parsed_mod: ItemMod = parse_macro_input!(foo_tokens as ItemMod);
// ...
}
Even this caveat can be removed if you make use of indirect imports (explained below), which are capable of working without requiring the token source to be a dependency of the target.
#[export_tokens]
You can apply the #[export_tokens]
macro to any
Item, with the exception of foreign modules,
impls, unnamed decl macros, and use declarations.
When you apply #[export_tokens]
to an item, a const
variable is generated immediately after
the item and set to a &'static str
containing the source code of the item. The const
variable is hidden from docs and its name consists of the upcased item name (i.e. the ident),
prefixed with __EXPORT_TOKENS__
, to avoid any collisions with any legitimate constants that
may have been defined.
This allows the tokens for the item to be imported using the import_tokens!
macro.
Optionally, you may specify a disambiguation path for the item as an argument to the macro, such as:
#[export_tokens(my::cool::ItemName)]
fn my_item () {
// ...
}
Any valid syn::TypePath
-compatible item is acceptable as input for #[export_tokens]
and
this input is optional. Furthermore the path need not exist -- you just have to use the same
path when you import.
Expansion
#[export_tokens]
fn foo_bar(a: u32) -> u32 {
a * 2
}
expands to:
#[allow(dead_code)]
fn foo_bar(a: u32) -> u32 {
a * 2
}
#[allow(dead_code)]
#[doc(hidden)]
pub const __EXPORT_TOKENS__FOO_BAR: &'static str = "fn foo_bar(a : u32) -> u32 { a * 2 }";
NOTE: items marked with #[export_tokens]
do not need to be public, however they do need to be
in a module that is accessible from wherever you intend to call import_tokens!
.
import_tokens!
You can pass the path of any item that has had the #[export_tokens]
attribute applied to it
directly to the import_tokens!
macro to get a
TokenStream2 of the
foreign item.
For example, suppose the foo_bar
function mentioned above is located in another crate and can
be accessed via really::cool::path::foo_bar
. As long as that path is accessible from the
current context (i.e. could be loaded via a use
statement if you wanted to), import_tokens!
will expand to a TokenStream2
of the item, e.g.:
let tokens = import_tokens!(cool::path::foo_bar);
This style of importing is called a direct import because we directly include the code we are exporting into the context where the tokens are being used (usually a proc macro crate).
Expansion
The example above would roughly expand to:
let tokens = cool::path::__EXPORT_TOKENS__FOO_BAR.parse::<TokenStream2>().unwrap();
import_tokens_indirect!
While direct imports are useful, there are situations where it would be impractical or extremely cumbersome to have the crate where your tokens are exported from (i.e. the "source" crate) be a dependency of your proc macro crate where those tokens are used (i.e. the "target crate"). This is especially true in scenarios where your proc macro crate is consumed by arbitrary downstream users who cannot modify your proc macro crate in any way without forking it. We provide a workaround via what we call "indirect imports". Another use-case for indirect imports is scenarios where the item in question is hidden behind a private module, as indirect imports can work around this scenario.
Calling import_tokens_indirect!
is slightly different from calling import_tokens!
in that
indirect imports will work even when the item whose tokens you are importing is contained in a
crate that is not a dependency of the current crate, so long as the following requirements are
met:
- The "indirect" feature must be enabled for
macro_magic
, otherwise theimport_tokens_indirect!
macro will not be available. - The source crate and the target crate must be in the same cargo workspace. This is a non-negotiable hard requirement when using indirect imports, however direct imports will work fine across workspace boundaries (they just have other stricter requirements that can be cumbersome).
- The source crate and the target crate must both use the same version of
macro_magic
(this is not a hard requirement, but undefined behavior could occur with mixed versions). - Both the source crate and target crate must be included in the compilation target of the
current workspace such that they are both compiled. Unlike with direct imports, where you
explictily
use
the source crate as a dependency of the target crate, there needs to be some reason to compile the source crate, or its exported tokens will be unavailable. - The export path declared by the source crate must exactly match the path you try to import
in the target crate. If you don't manually specify an export path, then your import path
should be the name of the item that
#[export_tokens]
was attached to (i.e. theIdent
), however this approach is not recommended since you can run into collisions if you are not explicit about naming. For highly uniquely named items, however, this is fine. - The target crate must be a proc macro crate.
The vast majority of common use cases for macro_magic
meet these criteria, but if you run
into any issues where exported tokens can't be found, make sure your source crate is included
as part of the compilation target and that it is in the current workspace.
Keep in mind that you can use the optional attribute, #[export_tokens(my::path::Here)]
to
specify a disambiguation path for the tokens you are exporting. Otherwise the name of the item
the macro is attached to will be used, potentially causing collisions if you export items by
the same name from different contexts.
This situation will eventually be resolved when the machinery behind
caller_modpath is stabilized, which will allow
macro_magic
to automatically detect the path of the #[export_tokens]
caller.
A peculiar aspect of how #[export_tokens(some_path)]
works is the path you enter doesn't need
to be a real path. You could do #[export_tokens(completely::made_up::path::MyItem)]
in one
context and then import_tokens!(completely::made_up::path::MyItem)
in another context, and it
will still work as long as these two paths are the same. They need not actually exist, they are
just used for disambiguation so we can tell the difference between these tokens and other
potential exports of an item called MyItem
. The last segment does need to match the name of
the item you are exporting, however.
read_namespace!
Namespaces support (included as part of the "indirect" feature) allows you to group a number of
#[export_tokens]
calls and collect them into a Result<Vec<(String, TokenStream2)>>
with the
the read_namespace!(some::namespace)
macro, where the first component of the tuple
corresponds with the name of the item and the second component contains the tokens for that
item. The Result
is a std::io::Result
and any Err
variants that come back would indicate
an internal error (i.e. something tampered with the target
directory at an unexpected time)
or (more likely) that the specified namespace does not exist.
The #[export_tokens]
attribute automatically defines namespaces when you call it with an
argument. Namespaces function like directories, so if you define item A with the path
foo::bar::fizz
and item B with path foo::bar::buzz
, and you will get back both items if you
read the namespace foo::bar
i.e.:
let namespace_items = read_namespace!(foo::bar).unwrap();
let (name, tokens) = namespace_items.first().unwrap();
// name = "buzz"
// tokens = tokens for `foo::bar::buzz` item
Note that read_namespace!
always returns results sorted by name, so you can rely on the order
to be consistent.
re_export_tokens_const!
This macro allows you to re-export already exported tokens across modules and even crates. See the docs for more info.
Features
By default no features are enabled. The following features are supported:
Verbose
The "verbose" feature is disabled by default. If enabled, some extra debugging information will
be printed at compile-time indicating when files are written and read from the REFS_DIR
for
the purpose of debugging import_tokens_indirect!
.
Normal users of the crate should not need this feature, however it is quite useful if things go wrong for some reason.
Indirect
The "indirect" feature is disabled by default. When this feature is disabled, only
#[export_tokens]
, import_tokens!
and re_export_tokens_const!
will be available and the
read_namespace!
and import_tokens_indirect!
macros will not be compiled. When "indirect" is
enabled, all of these macros will be available and you will be able to do indirect imports and
read namespaces. Namespaces and indirect imports are only supported when the "indirect"
feature is enabled.
Overhead
Because the automatically generated constants created by #[export_tokens]
are only used in a
proc-macro context, these constants do not add any bloat to the final binary because they will
be optimized out in contexts where they are not used. Thus these constants are a zero-overhead
abstraction once proc-macro expansion completes. The same goes for the temporary files used by
the indirect imports approach. These artifacts only exist at compile time and do not make it
into the final binary.
On a micro-scale, direct imports are slightly more efficient than indirect imports because they
do not involve any extra IO activity, using only a const
to synchronize information between
source and target.
Safety
Direct imports via import_tokens!
are 100% safe and don't rely on anything sketchy about
compile-order or artifacts in the target
directory.
Indirect imports are also safe because of how the macro_magic
build script is constructed
(unlike macro_state
, which may stop working in the future depending on what changes are made
to the Rust language), however, under the hood indirect imports do rely on coordinating based
on files in the target
directory for the current workspace, so mileage may vary depending on
the context where you try to use this approach.
For this reason it is recommended to stick with import_tokens!
unless your use case requires
the extra flexibility provided by import_tokens_indirect!
. You can disable
import_tokens_indirect!
completely by not opting in to the "indirect" feature.
Dependencies
~2MB
~41K SLoC