8 unstable releases (3 breaking)
new 0.7.0 | May 1, 2024 |
---|---|
0.6.0 | Apr 26, 2024 |
0.5.1 | Apr 24, 2024 |
0.4.7 | Apr 20, 2024 |
#70 in Machine learning
717 downloads per month
11MB
205K
SLoC
ctranslate2-rs
This library provides Rust bindings for OpenNMT/CTranslate2. At this time, it has only been tested and confirmed to work on macOS and Linux. Windows support is available experimentally, but it has not been thoroughly tested and may have limitations or require additional configuration.
Compilation
If you plan to use GPU acceleration, CUDA and cuDNN are available.
Please enable the cuda
or cudnn
feature and set the CUDA_TOOLKIT_ROOT_DIR
environment variable appropriately.
Several backends are available for use: OpenBLAS, Intel MKL, Ruy, and Apple Accelerate.
- OpenBLAS: To use OpenBLAS, enable the
openblas
feature and add the path to the directory containinglibopenblas.a
to theLIBRARY_PATH
environment variable. - Intel MKL: To use Intel MKL, enable the
mkl
feature and set the path to the Intel libraries in theMKLROOT
environment variable (default is/opt/intel
). - Ruy: To use Ruy, enable the
ruy
feature. - Apple Accelerate: Available only on macOS, enable the
accelerate
feature to use Apple Accelerate.
If no feature is specified:
- On macOS, Apple Accelerate will be used by default.
- On Linux and Windows, Ruy will be used by default.
The installation of CMake is required to compile the library.
Additional notes for Windows:
it is necessary to add RUSTFLAGS=-C target-feature=+crt-static
to the environment variables for compilation.
Model Conversion for CTranslate2
To use model files with CTranslate2, they must first be converted.
Below is an example of how to convert the nllb-200-distilled-600M
model:
pip install ctranslate2
ct2-transformers-converter --model facebook/nllb-200-distilled-600M --output_dir nllb-200-distilled-600M --copy_files tokenizer.json
For more details, please refer to the CTranslate2's docs.
Example of text translation
The following example translates English to German and Japanese using the previously converted
model nllb-200-distilled-600M
.
use anyhow::Result;
use ct2rs::config::{Config, Device};
use ct2rs::{TranslationOptions, Translator};
use ct2rs::tokenizers::Tokenizer;
fn main() -> Result<()> {
let path = "/path/to/nllb-200-distilled-600M";
let t = Translator::new(&path, Tokenizer::new(&path)?, &Config::default())?;
let res = t.translate_batch_with_target_prefix(
&vec![
"Hello world!",
"This library provides Rust bindings for CTranslate2.",
],
&vec![vec!["deu_Latn"], vec!["jpn_Jpan"]],
&TranslationOptions {
return_scores: true,
..Default::default()
},
)?;
for r in res {
println!("{}, (score: {:?})", r.0, r.1);
}
Ok(())
}
Output
Hallo Welt!<unk>, (score: Some(-0.5597002))
このライブラリでは,CTranslate2 の Rust バインディングが提供されています., (score: Some(-0.56321025))
Example of text generation
use anyhow::Result;
use ct2rs::config::{Config, Device};
use ct2rs::{Generator, GenerationOptions};
use ct2rs::sentencepiece::Tokenizer;
fn main() -> Result<()> {
let path = "/path/to/model";
let g = Generator::new(&path, Tokenizer::new(&path)?, &Config::default())?;
let res = g.generate_batch(
&vec!["prompt"],
&GenerationOptions::default(),
)?;
for r in res {
println!("{:?}", r.0);
}
Ok(())
}
License
This application is released under the MIT License. For details, see the LICENSE file.
Dependencies
~21MB
~392K SLoC