Carbon

Carbon is a lightweight indexing framework on Solana. It provides a modular pipeline for sourcing data, decoding updates and processing them in order to build end-to-end indexers.

Components

Pipeline

The core of the framework. It orchestrates data flow from data sources through indexing pipes.

Datasources

A consumable datasource that will provide updates to the pipeline. These can either be AccountUpdate, TransactionUpdate or AccountDeletion.

Pipes

Process specific updates:

• Account Pipes handle account updates. Each contains an AccountDecoder and a Processor.
• Account Deletion Pipes handle account deletions. Each contains a Processor.
• Instruction Pipes handle transaction updates, instruction by instruction. Each contains an InstructionDecoder and a Processor.
• Transaction Pipes handle transaction updates, after schema-matching the whole transaction. Each contains a Schema and a Processor.

Metrics

Collect and report on pipeline performance and operational data.

Our premade metrics crates assist with common use cases:

Usage

Basic Setup

use carbon_core::pipeline::Pipeline;
use carbon_rpc_block_subscribe_datasource::{RpcBlockSubscribe, Filters};
use solana_client::{
    rpc_config::{RpcBlockSubscribeConfig, RpcBlockSubscribeFilter},
};
use crate::{
    MyAccountDecoder, MyAccountProcessor,
    MyInstructionDecoder, MyInstructionProcessor,
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let pipeline = Pipeline::builder()
        .datasource(
            RpcBlockSubscribe::new(
                env::var("RPC_URL")?,
                Filters::new(RpcBlockSubscribeFilter::MentionsAccountOrProgram(env::var("MY_PROGRAM_ID")?), None)
            )
        )
        .instruction(MyInstructionDecoder::new(), MyInstructionProcessor)
        .metrics(Arc::new(LogMetrics::new()))
        .build()?;

    pipeline.run().await?;

    Ok(())
}

Generating Decoders from IDL

Decoders implementations allow the pipeline to input raw account or instruction data and to receive deserialized account or instruction data. They are the backbone of indexing with Carbon.

Carbon provides a CLI tool to generate decoders based on IDL files (Anchor, Codama) or from a provided program address with a network specified to fetch an on-chain PDA IDL. This can significantly speed up the process of creating custom decoders for your Solana programs.

CLI Installation

Install the Carbon CLI via npm:

# Install globally
npm install -g @sevenlabs-hq/carbon-cli

# Or use npx (no installation required)
npx @sevenlabs-hq/carbon-cli

CLI Usage

carbon-cli parse [OPTIONS]
carbon-cli scaffold [OPTIONS]

Parse Options

• -i, --idl <fileOrAddress>: Path to an IDL json file or a Solana program address
• -o, --out-dir <dir>: Output directory for generated code
• -c, --as-crate: Generate as a Cargo crate layout
• -s, --standard <anchor|codama>: Specify the IDL standard to parse (default: anchor)
• --event-hints <csv>: Comma-separated names of defined types to parse as CPI Events (Codama only)
• -u, --url <rpcUrl>: RPC URL for fetching IDL when using a program address
• --no-clean: Do not delete output directory before rendering

Scaffold Options

• -n, --name <string>: Name of your project
• -o, --out-dir <dir>: Output directory
• -d, --decoder <name>: Decoder name (auto-detected from IDL)
• --idl <fileOrAddress>: IDL file or program address
• --idl-standard <anchor|codama>: IDL standard
• --idl-url <rpcUrl>: RPC URL for fetching IDL (when using program address)
• --event-hints <csv>: Event hints for Codama IDL
• -s, --data-source <name>: Name of data source
• -m, --metrics <log|prometheus>: Metrics to use (default: log)
• --with-postgres <boolean>: Include Postgres wiring and deps (default: true)
• --with-graphql <boolean>: Include GraphQL wiring and deps (default: true)
• --with-serde <boolean>: Include serde feature for decoder (default: false)
• --force: Overwrite output directory if it exists

Examples

Generate decoder from Anchor IDL:

carbon-cli parse --idl my_program.json --out-dir ./src/decoders

Generate decoder from program address:

carbon-cli parse --idl LBUZKhRxPF3XUpBCjp4YzTKgLccjZhTSDM9YuVaPwxo -u mainnet-beta --out-dir ./desired-folder

Scaffold project using IDL file:

carbon-cli scaffold --name my-project --out-dir ./desired-folder --idl ./idl.json --data-source yellowstone-grpc

For more detailed usage information and examples, check out the npm package documentation.

Implementing Processors

use carbon_core::account::{AccountDecoder, AccountMetadata, AccountProcessorInputType, DecodedAccount};
use crate::MyCustomAccountData;

struct MyAccountProcessor;

#[async_trait]
impl Processor for MyAccountProcessor {
    type InputType = AccountProcessorInputType<MyCustomAccountData>;

    async fn process(
        &mut self,
        input: Self::InputType,
        metrics: Arc<MetricsCollection>,
    ) -> CarbonResult<()> {
        // Implement processing logic
    }
}

Implementing a Datasource

For most use cases, we recommend choosing from one of our datasource crates:

You can still implement custom datasources in the following manner:

use carbon_core::datasource::{Datasource, Update, UpdateType};

struct MyDataSource;

#[async_trait]
impl Datasource for MyDataSource {
    async fn consume(
        &self,
        sender: &tokio::sync::mpsc::UnboundedSender<Update>,
        cancellation_token: CancellationToken,
    ) -> CarbonResult<()> {
        // Implement data fetching and sending logic
    }

    fn update_types(&self) -> Vec<UpdateType> {
        vec![UpdateType::AccountUpdate, UpdateType::Transaction]
    }
}

Available Program Decoders

Decoders for most popular Solana programs are published and maintained:

Pre-commit hooks

To activate the pre-commit hook, run ./.pre-commit.sh once. This will cause the following rules to be registered and called at the next git commit.

The following checks will be performed:

• fmt: Checks code formatting using cargo fmt --check.
• clippy: Runs clippy on the codebase to catch potential issues.
• cargo_sort: Utilizes cargo-sort to ensure Cargo.toml files are sorted correctly.
• machete: Checks for unused Cargo dependencies using cargo-machete.

Test

cargo test

License

We are under the MIT license.