MotherDuck Sync

A Rust CLI for syncing PostgreSQL data to MotherDuck for analytics.

PostgreSQL/Supabase  →  motherduck-supasync  →  MotherDuck (DuckDB)

Features

• Incremental sync — Only sync new/changed records
• Batch processing — ~10k records/sec throughput
• Column mapping — Rename columns between source and target
• Privacy-first — Schema details stored in secrets, not code
• CI/CD ready — GitHub Actions workflow included

Quick Start

# Install
cargo install --path .

# Set credentials
export DATABASE_URL="postgres://...@pooler.supabase.com:6543/postgres"
export MOTHERDUCK_TOKEN="your_token"
export SYNC_TABLES_JSON='[{"source":"users","target":"users","pk":["id"]}]'

# Sync
motherduck-supasync sync

Documentation

Configuration

1. Create Table Config

cp tables.example.json tables.local.json

[
  {
    "source": "staging_users",
    "target": "full_users",
    "pk": ["id"],
    "columns": ["id", "email", "created_at"],
    "order_by": "created_at"
  }
]

2. Generate Secret

motherduck-supasync generate-secret --input tables.local.json

3. Set GitHub Secrets

Commands

motherduck-supasync sync              # Incremental sync
motherduck-supasync sync --full       # Full resync
motherduck-supasync test              # Test connectivity
motherduck-supasync status            # Show unsynced counts
motherduck-supasync query --counts    # Query MotherDuck
motherduck-supasync generate-secret   # Generate config secret

Table Config Fields

Architecture

┌──────────────┐     ┌─────────────────┐     ┌─────────────┐
│  PostgreSQL  │────▶│ motherduck-supasync │────▶│ MotherDuck  │
└──────────────┘     └─────────────────┘     └─────────────┘
       │                     │
       └─────────────────────┘
              Mark synced

1. Fetch unsynced records (WHERE synced_to_motherduck = false)
2. Batch insert to MotherDuck (1000 records/batch)
3. Mark source records as synced

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT License (LICENSE-MIT)

at your option.

Contributing

Contributions welcome! Please read the license terms before contributing.