2 releases

✓ Uses Rust 2018 edition

0.0.2 Jun 28, 2020
0.0.1 Nov 14, 2019

#24 in Machine learning

MIT/Apache

11MB
234K SLoC

C++ 226K SLoC // 0.1% comments Rust 5K SLoC // 0.1% comments C 2.5K SLoC // 0.2% comments JavaScript 468 SLoC // 0.7% comments Python 132 SLoC // 0.5% comments Shell 130 SLoC // 0.4% comments Batch 56 SLoC // 0.3% comments

License-MIT/Apache-2.0 Rust

autograph

Machine Learning Library for Rust

Features

  • Safe API
  • Thread Safe
  • CPU and CUDA are fully supported
  • Flexible (Dynamic Backward Graph)

Layers

  • Dense
  • Conv2d
  • MaxPool2d
  • Relu

Loss Functions

  • CrossEntropyLoss

Datasets

  • MNIST

Graphs

During training, first a forward pass is run to determine the outputs of a model and compute the loss. Then a backward pass is run to compute the gradients which are used to update the model parameters. Autograph constructs a graph of operations in the forward pass that is then used for the backward pass. This allows for intermediate values and gradients to be lazily allocated and deallocated using RAII in order to minimize memory usage. Native control flow like loops, if statements etc. can be used to define a forward pass, which does not need to be the same each time. This allows for novel deep learning structures like RNN's and GAN's to be constructed, without special hardcoding.

Extension

See branch extend_api. This branch adds feature xapi which enables additional methods needed to add ops to autograph. This is experimental and feedback welcome. Currently only supports cpu operations. Autograph is highly modularized, cpu operations can be implemented using pure Rust, or via c/c++ to oneDNN (see https://oneapi-src.github.io/oneDNN/), or some other means. The xapi feature provides access to the dnnl::engine and dnnl::stream needed to perform operations with dnnl (now oneDNN).

Supported Platforms

Tested on Ubuntu-18.04, Windows Server 2019, and macOS Catalina 10.15. Generally you will want to make sure that OpenMP is installed. Currently cmake / oneDNN has trouble finding OpenMP on mac and builds in sequential mode. This greatly degrades performance (approx 10x slower) but it will otherwise run without issues. If you have trouble building autograph, please create an issue.

CUDA

Cuda can be enabled by passing the feature "cuda" to cargo. CUDA https://developer.nvidia.com/cuda-downloads and cuDNN https://developer.nvidia.com/cudnn must be installed. See https://github.com/bheisler/RustaCUDA and https://github.com/charles-r-earp/cuda-cudnn-sys for additional information.

Datasets

Autograph includes a datasets module enabled with the features datasets. This currently has the MNIST dataset, which is downloaded and saved automatically. The implementation of this is old and outdated (it uses reqwest among others which now uses async), and compiles slowly. Potentially overkill for such a small dataset, but for adding new datasets (like ImageNet), we will need an updated, fast implementation.

Getting Started

If you are new to Rust, you can get it and find documentation here: https://www.rust-lang.org/ View the documentation by:

cargo doc --open [--features "[datasets] [cuda]"]

To add autograph to your project add it as a dependency in your cargo.toml (features are optional):

[dependencies]
autograph = { version = 0.0.2, features = ["datasets", "cuda"] }

Tests

Run the unit-tests with (passing the feature cuda additionally runs cuda tests):

cargo test --lib [--features cuda]

Examples

Run the examples with:

cargo run --example [example] --features "datasets [cuda]" --release

See the examples directory for the examples.

Benchmarks

Run the benchmarks with:

cargo bench [--features cuda]

Roadmap

  • Optimizers (SGD, Adam)
  • Saving and loading of models / Serde
  • Data transfers between devices (local model parallel)
  • Data Parallel (multi-gpu)
  • Remote Distributed Parallel (ie training on multiple machines)

Dependencies

~3.5–7MB
~159K SLoC