crates_llms_txt

An asynchronous Rust library for retrieving Rust crates documentation structure and LLM configurations.

Features

• Asynchronously fetch crates documentation index
• Parse documentation structure into Rust structs
• Generate standard LLM configurations, including sessions and full_sessions

lib.rs:

Crates LLMs TXT

A Rust library for extracting and processing Rust crate documentation from both online sources (docs.rs) and local crates. This library provides functionality to fetch, parse, and structure rustdoc JSON data for use in LLM training or other documentation processing tasks.

Features

• Online Documentation: Fetch documentation from docs.rs with automatic zstd decompression
• Local Documentation: Generate documentation from local Cargo projects
• Version Compatibility: Handle different rustdoc JSON format versions automatically
• Feature Control: Generate docs with specific feature sets or all features
• Flexible Output: Structured data suitable for LLM training or analysis

Examples

Fetching Online Documentation

use crates_llms_txt::CrateDocs;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Fetch latest version
    let docs = CrateDocs::from_online("serde", None).await?;
    
    // Fetch specific version
    let docs = CrateDocs::from_online("clap", Some("4.5.39".to_string())).await?;
    
    println!("Found {} documentation items", docs.sessions.len());
    Ok(())
}

Generating Local Documentation

use std::path::PathBuf;
use crates_llms_txt::CrateDocs;

// Generate docs with all features
let docs = CrateDocs::from_local(
    PathBuf::from("./Cargo.toml"),
    Some("stable".to_string()),
)?;

// Generate docs with specific features
let docs = CrateDocs::from_local_with_features(
    PathBuf::from("./Cargo.toml"),
    false, // don't disable default features
    Some(vec!["async".to_string()]),
    None, // auto-detect toolchain
)?;