2 releases
0.1.1 | Aug 10, 2024 |
---|---|
0.1.0 | Jul 30, 2024 |
#182 in Science
59 downloads per month
33KB
544 lines
JYAFN extensions
JYAFN extensions are useful for adding any extra capabilities to your jyafn
code that the standard implementation cannot provide. The primary usecase is when one has very complicated thrid-party code which is not compatible with jyafn
(e.g., makes calls to native functions). This code normally could be reimplmemented in jyafn
, but it would take time and money. If this code also happens to have an implementation which is not dependent (or minimally dependent) on Python, it's normally straightforward to create a C wrapper around it and expose it to JYAFN using the extension API. A prime example of this kind of code is the LightGBM library, which has its extension implemented in the extensions
folder.
This crate provides a way to write extensions in idiomatic Rust without unsafe code at all. It will use Rust macros to generate all the boilerplate code and take care of managing all the unsafety in a correct way.
What are extensions?
Extensions are simply shared objects that conform to a given C API. These shared objects are stored somewhere in the filesystem, by default in ~/.jyafn/extensions
and more generally in the JYAFN_PATH
environment variable. They are then loaded by jyafn whenever a new computational graph declares an resource that depends on a resource type provided by the extension. As such, extensions come with some big expectations:
- Extensions are trusted code. They don't operate in the same limited and sandboxed environment of
jyafn
functions. They can run anything, from awesome intrincate algorithms to bugs to viruses. - Extensions are dependencies and dependencies have to be managed. They need to be installed beforehand from somewhere, otherwise the graphs depending on them will not work.
- Extensions are system-dependent. There is no code once, run everywhere. They need to be compiled for every architecture.
Therefore, extensions should have their use kept to a minimum and should be reseved only for highly-reusable code.
What are resources?
Resources are, simply put, just like objects in Python. They can be created and can be accessed using methods. Unlike Python objects, they are made to be stricty immutable and have to be serializable (i.e., represented as bytes in a file). An extension can declare a list of resources, each of which can declare a list of methods. A resource has a simple lifecycle:
- It's created from some external binary data representation.
- It's queried upon with methods at every method call in the
jyafn
function. - It's serialized into bytes to be put in a file together with the rest of the
jyafn
graph, potentially to be read and recreated in another machine.
Thus, we have a very simple interface for creating resources:
from_bytes
: builds a new resource from serialized binary data.dump
: represents a resource as serialized data.get_method
: gets information on a method (its function pointer, input and output layout) for the execution of a call.
A last method, size
, is also to be implemented to keep track of heap consumption of each resource.
Where to go from here
If you intend to roll out your own extension, you can check out the extensions
folder for sample implementations. The dummy
extension has a basic, simple example of a very simple extension, while the lightgbm
extension shows a fully operational extension for the LightGBM library.
However, if you, as most people, are looking for information of how to work with extensions as a user, check out the "Playing with extensions" (implementation pending) to learn more.
Dependencies
~0.7–1.6MB
~35K SLoC