#llm #ai #openai #huggingface

app llmvm-outsource

An llmvm backend which sends text and chat generation requests to known hosted language model providers

2 stable releases

1.1.0 Nov 14, 2023
1.0.0 Aug 9, 2023

#48 in Machine learning

36 downloads per month

MPL-2.0 license

88KB
1.5K SLoC

llmvm-outsource

Crates.io GitHub

An llmvm backend which sends text and chat generation requests to known hosted language model providers.

Supported providers:

  • OpenAI (text & chat interface)
  • Hugging Face (text interface)
  • Ollama

Example of an llmvm model ID for this backend: outsource/openai-chat/gpt-3.5-turbo

Installation

Install this backend using cargo.

cargo install llmvm-outsource

Usage

The backend can either be invoked directly, via llmvm-core or via a frontend that utilizes llmvm-core.

To invoke directly, execute llmvm-outsource -h for details.

llmvm-outsource http can be invoked to create a HTTP server for remote clients.

Configuration

Run the backend executable to generate a configuration file at:

  • Linux: ~/.config/llmvm/outsource.toml.
  • macOS: ~/Library/Application Support/com.djandries.llmvm/outsource.toml
  • Windows: AppData\Roaming\djandries\llmvm\config\outsource.toml
Key Required? Description
openai_api_key If using OpenAI API key for OpenAI requests.
huggingface_api_key If using Hugging Face API key for Hugging Face requests.
ollama_endpoint If using Ollama Endpoint for Ollama requests (defaults to http://127.0.0.1:11434/api/generate)
tracing_directive No Logging directive/level for tracing
stdio_server No Stdio server settings. See llmvm-protocol for details.
http_server No HTTP server settings. See llmvm-protocol for details.

Hugging Face custom endpoints

Custom hosted endpoints may be used by supplying the prefix endpoint=, followed by the endpoint URL in the model name component of the model ID.

For example, the model ID could be outsource/huggingface-text/endpoint=https://yourendpointhere.

License

Mozilla Public License, version 2.0

Dependencies

~15–28MB
~444K SLoC