60 releases
0.1.67 | Aug 9, 2023 |
---|---|
0.1.66 | Aug 8, 2023 |
0.1.58 | Jul 31, 2023 |
0.1.28 | Jun 30, 2023 |
#283 in Science
497 downloads per month
2MB
44K
SLoC
rs-llama-cpp
Automated Rust bindings generation for LLaMA.cpp
Description
LLaMA.cpp is under heavy development with contributions pouring in from numerous individuals every day. Currently, its C API is very low-level and given how fast the project is evolving, keeping up with the changes and porting the examples into a higher-level API prove to be difficult. As a trade-off, this project prioritizes automation over flexibility by automatically generating Rust bindings for the main example of LLaMA.cpp.
Limitations
The main design goal of this project is to minimize the effort of updating LLaMA.cpp by automating as many steps as possible. However, this approach does have some limitations:
- The API is very high-level, resembling a call to the main function of LLaMA.cpp and receiving tokens through a callback function.
- Currently, the project does not expose parameters with types that are more challenging to convert to Rust, such as
std::unordered_map
andstd::vector
. - Some of the parameters exposed via the Rust API are only relevant for a CLI.
- The generated C++ library outputs a significant amount of debug information to
stderr
andstdout
. This is not configurable at the moment
Usage
use rs_llama_cpp::{gpt_params_c, run_inference, str_to_mut_i8};
fn main() {
let params: gpt_params_c = {
gpt_params_c {
model: str_to_mut_i8("/path/to/model.bin"),
prompt: str_to_mut_i8("Hello "),
..Default::default()
}
};
run_inference(params, |token| {
println!("Token: {}", token);
if token.ends_with("\n") {
return false; // stop inference
}
return true; // continue inference
});
}