#infa #machine #learning #llama

bin+lib infa-safetensors

A minimal rust machine learning library in wip

1 unstable release

0.0.1 Oct 10, 2024

#2 in #infa

MIT license

18KB
487 lines

infa

Rust + CUDA = Fast and simple inference library from scratch

requirements

Linux computer with CUDA 12~, cublas, rust installed. You need at least sm_80 micro architecture. (This is hardcoded for now.)

compared to pytorch and llama.cpp

WIP

roadmap

Our first goal is to support bloat16 Llama 3.2 1B inference.

Dependencies

~1.1–2.1MB
~43K SLoC