triton-distributed

Overview

This repository contains the core components of a distributed GenAI inference framework written in Rust.

Features:

• A High level API defined in component.rs to build distributed applications.
• A Distributed runtime to manage the distributed execution of the inference graph.
• Uses NATS for component communication and etcd for service discovery, allowing engines to be distributed across multiple nodes while maintaining a unified processing graph.
• Modular design makes it easy to build inference pipelines by composing reusable components.

Install Rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Rust Installation Guide

Rust Build

cargo build

Run Tests

cargo test