edgequake-pdf2md

Convert PDF documents to clean Markdown using Vision Language Models

edgequake-pdf2md is a Rust CLI and library that converts PDF files (local or URL) into well-structured Markdown using vision-capable LLMs. It rasterises each page with pdfium, sends the image to a VLM (GPT-4.1, Claude, Gemini, etc.), and post-processes the result into clean Markdown.

Inspired by pyzerox, rebuilt in Rust for speed and reliability.

Features

• Multi-provider — AWS Bedrock (default), OpenAI, Anthropic, Google Gemini, Mistral AI, Azure, Ollama, or any OpenAI-compatible endpoint
• Fast — concurrent page processing with configurable parallelism
• Accurate — 10-rule post-processing pipeline fixes tables, removes hallucinations, normalises output
• Flexible — page selection, fidelity tiers, custom system prompts, streaming API
• Self-contained — pdfium (~5 MB) embedded in the binary by default; no runtime downloads, no env vars
• Cross-platform — macOS (arm64/x64), Linux (x64/aarch64), Windows (x64/arm64)
• Library + CLI — use as a Rust crate or standalone command-line tool

Quick Start

Self-contained binary — zero runtime setup. Starting from v0.4.0, the PDFium engine (~5 MB) is embedded inside the binary at compile time (bundled feature is now the default). No download required at runtime, no DYLD_LIBRARY_PATH, no environment variables needed.

Build-time auto-download: if you don't set PDFIUM_BUNDLE_LIB, the correct pdfium library is downloaded automatically during cargo build and cached in ~/.cargo/pdfium-bundle/. Use PDFIUM_LIB_PATH to point to an existing copy at runtime (download mode, without bundled feature).

1. Set credentials

# AWS Bedrock (recommended — cheapest, default provider)
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_REGION="eu-west-1"        # optional, default: us-east-1
# or
export OPENAI_API_KEY="sk-..."       # OpenAI
# or
export ANTHROPIC_API_KEY="sk-ant-..."  # Anthropic
# or
export GEMINI_API_KEY="AI..."          # Google Gemini
# or
export MISTRAL_API_KEY="..."           # Mistral AI (uses pixtral-12b-2409)

2. Build & run

cargo build --release

# Convert a PDF
./target/release/pdf2md document.pdf -o output.md

# Convert from URL
./target/release/pdf2md https://arxiv.org/pdf/1706.03762 -o paper.md

# Inspect metadata (no API key needed)
./target/release/pdf2md --inspect-only document.pdf

Or install globally:

cargo install edgequake-pdf2md
pdf2md document.pdf -o output.md

How It Works

PDF ──▶ pdfium ──▶ PNG images ──▶ base64 ──▶ VLM API ──▶ post-process ──▶ Markdown
        render      per page       encode     (concurrent)   10 rules       assembled

1. Input — resolve local file or download from URL
2. Render — rasterise pages to images via pdfium-render
3. Encode — base64-encode each page image
4. VLM — send images to a vision LLM with a structured system prompt
5. Post-process — strip fences, fix tables, remove hallucinated images, normalise whitespace
6. Assemble — join pages with optional separators and YAML front-matter

See docs/how-it-works.md for the full pipeline walkthrough with diagrams.

Usage

# Basic conversion (uses AWS Bedrock by default)
pdf2md document.pdf -o output.md

# Specific pages
pdf2md --pages 1-5 document.pdf -o first_five.md

# High fidelity with a better model
pdf2md --fidelity tier3 --model gpt-4.1 --provider openai --dpi 200 paper.pdf -o paper.md

# Consistent formatting across pages (sequential mode)
pdf2md --maintain-format --separator hr book.pdf -o book.md

# JSON output with metadata
pdf2md --json --metadata document.pdf > output.json

# Use a different Bedrock model
pdf2md --provider bedrock --model amazon.nova-pro-v1:0 document.pdf

# Use Anthropic
pdf2md --provider anthropic --model claude-sonnet-4-20250514 document.pdf

# Use Mistral (pixtral-12b-2409 auto-selected as vision model)
export MISTRAL_API_KEY=your-key
pdf2md document.pdf
# or explcitly:
pdf2md --provider mistral --model pixtral-12b-2409 document.pdf

# Use local Ollama
pdf2md --provider ollama --model llava document.pdf

# Resumable conversion with checkpoints (v0.7)
pdf2md --checkpoint-dir ./checkpoints big-doc.pdf -o out.md

# Resume after interruption (re-run the same command)
pdf2md --checkpoint-dir ./checkpoints big-doc.pdf -o out.md

# Force fresh conversion, clearing existing checkpoints
pdf2md --checkpoint-dir ./checkpoints --no-resume big-doc.pdf -o out.md

Run pdf2md --help for the full reference, including supported models and cost estimates.

See docs/examples.md for more usage patterns.

Supported Providers & Models

Cost estimate: A 50-page document costs ~$0.01 with amazon.nova-lite-v1:0, ~$0.02 with gpt-4.1-nano.

See docs/providers.md for detailed comparisons, cost calculators, and selection guide.

Library Usage

Add to your Cargo.toml:

[dependencies]
edgequake-pdf2md = "0.7"
tokio = { version = "1", features = ["full"] }

Basic conversion

use edgequake_pdf2md::{convert, ConversionConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ConversionConfig::builder()
        .model("amazon.nova-lite-v1:0")
        .provider_name("bedrock")
        .pages(edgequake_pdf2md::PageSelection::Range(1, 5))
        .build()?;

    let output = convert("document.pdf", &config).await?;
    println!("{}", output.markdown);
    println!("Processed {}/{} pages", output.stats.processed_pages, output.stats.total_pages);
    Ok(())
}

Convert PDF bytes in memory (v0.2)

No temp-file management needed — pass raw bytes directly:

use edgequake_pdf2md::{convert_from_bytes, ConversionConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let bytes = std::fs::read("document.pdf")?;  // or from DB / network
    let config = ConversionConfig::default();
    let output = convert_from_bytes(&bytes, &config).await?;
    println!("{}", output.markdown);
    Ok(())
}

Per-page progress callbacks (v0.2)

use edgequake_pdf2md::{convert, ConversionConfig, ConversionProgressCallback};
use std::sync::Arc;

struct MyProgress;

impl ConversionProgressCallback for MyProgress {
    fn on_conversion_start(&self, total: usize) {
        eprintln!("Starting conversion of {total} pages");
    }
    fn on_page_complete(&self, page: usize, total: usize, chars: usize) {
        eprintln!("  ✓ Page {page}/{total} — {chars} chars");
    }
    fn on_page_error(&self, page: usize, total: usize, error: String) {
        eprintln!("  ✗ Page {page}/{total} failed: {error}");
    }
    fn on_conversion_complete(&self, total: usize, success: usize) {
        eprintln!("Done: {success}/{total} pages converted");
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ConversionConfig::builder()
        .progress_callback(Arc::new(MyProgress) as Arc<dyn ConversionProgressCallback>)
        .build()?;
    let output = convert("document.pdf", &config).await?;
    println!("{}", output.markdown);
    Ok(())
}

Strict error on partial failure (v0.2)

By default, page failures are non-fatal. Use into_result() to promote them to errors:

use edgequake_pdf2md::{convert, ConversionConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ConversionConfig::default();
    // into_result() returns Err(PartialFailure) if any pages failed
    let output = convert("document.pdf", &config).await?.into_result()?;
    println!("{}", output.markdown);
    Ok(())
}

Resumable conversions with checkpoints (v0.7)

Enable page-level checkpointing for large documents (500–2000+ pages). If a conversion is interrupted, re-running with the same settings resumes from where it left off — already-completed pages are loaded instantly from the checkpoint store, skipping render + VLM calls entirely.

use edgequake_pdf2md::{convert, ConversionConfig, FileCheckpointStore};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = Arc::new(FileCheckpointStore::new("./checkpoints"));

    let config = ConversionConfig::builder()
        .checkpoint_store(store)
        .build()?;

    let output = convert("big-document.pdf", &config).await?;
    println!(
        "Processed {} pages ({} resumed from checkpoint)",
        output.stats.processed_pages,
        output.stats.resumed_pages,
    );
    Ok(())
}

Checkpoints are keyed by a deterministic conversion ID derived from the PDF content, provider, model, fidelity, and DPI. Changing any setting creates a separate checkpoint set. Checkpoints are automatically cleared when all pages succeed.

Provider injection (v0.2)

Pass a pre-built Arc<dyn LLMProvider> directly — useful for sharing providers across multiple conversions and for testing with mocks:

use edgequake_pdf2md::{convert, ConversionConfig};
use edgequake_llm::ProviderFactory;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (provider, _) = ProviderFactory::from_env()?;
    let config = ConversionConfig::builder()
        .provider(Arc::clone(&provider))   // injected; highest priority
        .build()?;
    let output = convert("document.pdf", &config).await?;
    println!("{}", output.markdown);
    Ok(())
}

Provider resolution order (highest-to-lowest priority):

1. config.provider — explicit Arc<dyn LLMProvider> injection
2. config.provider_name + config.model — named provider
3. EDGEQUAKE_LLM_PROVIDER + EDGEQUAKE_MODEL environment variables
4. Auto-detect from credentials (AWS_ACCESS_KEY_ID, OPENAI_API_KEY, ANTHROPIC_API_KEY, MISTRAL_API_KEY, …)

Also available: streaming API (convert_stream, convert_stream_from_bytes), sync wrapper (convert_sync), metadata inspection (inspect).

See API docs on docs.rs for the full API reference.

Configuration

All options can be set via CLI flags, environment variables, or the builder API:

See docs/configuration.md for the complete reference.

Development

# Setup
make setup          # Check pdfium + API key

# Build
make build          # Release binary
make build-dev      # Debug binary

# Test
make test           # Unit tests (no API key needed)
make test-e2e       # Integration tests (needs API key)
make test-all       # All tests

# Quality
make lint           # Clippy
make fmt            # Format code
make ci             # format + lint + unit tests

# Try it
make demo           # Convert sample page
make inspect-all    # Inspect test PDFs

Documentation

Dependencies

Python users: edgequake-litellm v0.1.3 (PyPI) is a drop-in LiteLLM replacement backed by edgequake-llm. It supports Azure OpenAI via model="azure/<deployment>".

External References

• pdfium — Google's open-source PDF rendering engine
• pdfium-binaries — Pre-built pdfium binaries for all platforms
• pyzerox — The Python project that inspired this tool
• OpenAI Vision API — Image understanding with GPT-4.1
• Anthropic Vision — Image understanding with Claude
• Google Gemini — Vision capabilities

License

Copyright 2026 Raphaël MANSUY

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

See LICENSE for the full text.