5 unstable releases

0.3.0 Oct 12, 2023
0.2.0 Aug 29, 2023
0.1.2 Aug 3, 2023
0.1.1 Jul 20, 2023
0.1.0 Jul 10, 2023

#307 in FFI

38 downloads per month

Custom license

3.5MB
65K SLoC

C++ 28K SLoC // 0.1% comments C 21K SLoC // 0.1% comments CUDA 6K SLoC // 0.0% comments Python 5K SLoC // 0.1% comments Metal Shading Language 2K SLoC // 0.0% comments Objective-C 1K SLoC // 0.1% comments Shell 1K SLoC // 0.1% comments Rust 815 SLoC // 0.0% comments JavaScript 327 SLoC // 0.1% comments RPM Specfile 163 SLoC // 0.2% comments Vim Script 132 SLoC // 0.1% comments Zig 112 SLoC // 0.0% comments Swift 53 SLoC Batch 48 SLoC INI 5 SLoC

rust_llama.cpp

Docs Crates.io

LLama.cpp rust bindings.

The rust bindings are mostly based on https://github.com/go-skynet/go-llama.cpp/

Building Locally

Note: This repository uses git submodules to keep track of LLama.cpp.

Clone the repository locally:

git clone --recurse-submodules https://github.com/mdrokz/rust-llama.cpp
cargo build

Usage

[dependencies]
llama_cpp_rs = "0.2.0"
use llama_cpp_rs::{
    options::{ModelOptions, PredictOptions},
    LLama,
};

fn main() {
    let model_options = ModelOptions::default();

    let llama = LLama::new(
        "../wizard-vicuna-13B.ggmlv3.q4_0.bin".into(),
        &model_options,
    )
    .unwrap();

    let predict_options = PredictOptions {
        token_callback: Some(Box::new(|token| {
            println!("token1: {}", token);

            true
        })),
        ..Default::default()
    };

    llama
        .predict(
            "what are the national animals of india".into(),
             predict_options,
        )
        .unwrap();
}

Examples

The examples contain dockerfiles to run them

see examples

TODO

  • Implement support for cublas,openBLAS & OpenCL #7
  • Implement support for GPU (Metal)
  • Add some test cases
  • Support for fetching models through http & S3
  • Sync with latest master & support GGUF
  • Add some proper examples https://github.com/mdrokz/rust-llama.cpp/pull/7

LICENSE

MIT

Dependencies