2 stable releases
| 1.1.0 | Nov 14, 2023 |
|---|---|
| 1.0.0 | Aug 9, 2023 |
#48 in Machine learning
36 downloads per month
88KB
1.5K
SLoC
llmvm-outsource
An llmvm backend which sends text and chat generation requests to known hosted language model providers.
Supported providers:
- OpenAI (text & chat interface)
- Hugging Face (text interface)
- Ollama
Example of an llmvm model ID for this backend: outsource/openai-chat/gpt-3.5-turbo
Installation
Install this backend using cargo.
cargo install llmvm-outsource
Usage
The backend can either be invoked directly, via llmvm-core or via a frontend that utilizes llmvm-core.
To invoke directly, execute llmvm-outsource -h for details.
llmvm-outsource http can be invoked to create a HTTP server for remote clients.
Configuration
Run the backend executable to generate a configuration file at:
- Linux:
~/.config/llmvm/outsource.toml. - macOS:
~/Library/Application Support/com.djandries.llmvm/outsource.toml - Windows:
AppData\Roaming\djandries\llmvm\config\outsource.toml
| Key | Required? | Description |
|---|---|---|
openai_api_key |
If using OpenAI | API key for OpenAI requests. |
huggingface_api_key |
If using Hugging Face | API key for Hugging Face requests. |
ollama_endpoint |
If using Ollama | Endpoint for Ollama requests (defaults to http://127.0.0.1:11434/api/generate) |
tracing_directive |
No | Logging directive/level for tracing |
stdio_server |
No | Stdio server settings. See llmvm-protocol for details. |
http_server |
No | HTTP server settings. See llmvm-protocol for details. |
Hugging Face custom endpoints
Custom hosted endpoints may be used by supplying the prefix endpoint=, followed by the endpoint
URL in the model name component of the model ID.
For example, the model ID could be outsource/huggingface-text/endpoint=https://yourendpointhere.
License
Dependencies
~15–28MB
~444K SLoC