Lib.rs

Devgen Splitter is a Rust library that breaks down source code into contextual chunks. It utilizes tree-sitter to identify code entities (such as classes, functions, and methods) and generate chunks with contextual metadata.

Features

• Language-aware code splitting
• Generate chunks with contextual metadata
• Support for multiple programming languages

why devgen-splitter?

If you are building a code search agent, you may want to the LLM to generate the link for related class, struct, enum, etc. Devgen Splitter can help you generate the chunks with contextual metadata.

Usage

Add devgen-splitter to your project:

cargo add devgen-splitter

Basic usage example:

use devgen_splitter::{SplitOptions, split};
let code = "fn main() { println!(\"Hello, world!\"); }";
let options = SplitOptions { chunk_line_limit: 10};
let chunks = split("example.rs", code, &options).unwrap();
for chunk in chunks {
    println!("Chunk: {:?}", chunk);
}

For more examples, go to examples

Supported Languages

More languages coming soon!

Language Mapping

The following table shows how different code structures are represented across various programming languages and their corresponding tree-sitter query rule names:

Development Status

Devgen Splitter is in active development. We welcome community contributions and feedback.