Lib.rs

Utilities to simplify running auto-regressive RTen models such as transformer decoders.

For working examples, see the examples in the rten-examples crate which import rten_generate.

rten-generate is a layer on top of RTen which handles the generation loop for auto-regressive transformer models (aka. "transformer decoders" or "generative AI"). This includes managing the KV cache, sampling and post-processing logits etc.