Lib.rs

Cargo Features

[dependencies]
llama_cpp = { version = "0.3.2", default-features = false, features = ["compat", "native", "avx", "avx2", "avx512", "avx512_vmbi", "avx512_vnni", "fma", "f16c", "accel", "mpi", "cuda", "cuda_f16", "cuda_dmmv", "cuda_mmq", "metal", "blas", "hipblas", "clblast", "vulkan", "sys_verbosity"] }

default = compat, native

These default features are set whenever llama_cpp is added without default-features = false somewhere in the dependency tree.

compat default

this feature modifies the symbols exposed by the generated libraries to avoid conflicts

Enables compat of llama_cpp_sys

native default = accel, avx, avx2, f16c, fma

Enables native of llama_cpp_sys

avx native

Enables avx of llama_cpp_sys

avx2 native

Enables avx2 of llama_cpp_sys

avx512

Enables avx512 of llama_cpp_sys

avx512_vmbi

Enables avx512_vmbi of llama_cpp_sys

avx512_vnni

Enables avx512_vnni of llama_cpp_sys

fma native

Enables fma of llama_cpp_sys

f16c native

implied when compiled using MSVC with avx2/avx512

Enables f16c of llama_cpp_sys

accel native

Accelerate framework

Enables accel of llama_cpp_sys

mpi

Enables mpi of llama_cpp_sys

cuda cuda_dmmv? cuda_f16? cuda_mmq?

Enables cuda of llama_cpp_sys

cuda_f16 = cuda

Enables cuda_f16 of llama_cpp_sys

cuda_dmmv = cuda

use dmmv instead of mmvq CUDA kernels

Enables cuda_dmmv of llama_cpp_sys

cuda_mmq = cuda

use mmq kernels instead of cuBLAS

Enables cuda_mmq of llama_cpp_sys

metal

Enables metal of llama_cpp_sys

blas

Enables blas of llama_cpp_sys

hipblas

Enables hipblas of llama_cpp_sys

clblast

Enables clblast of llama_cpp_sys

vulkan

Enables vulkan of llama_cpp_sys

sys_verbosity

increase sys crate tracing log verbosity level