DB-Sync Database Backup Synchronization System

A secure and stable database backup file synchronization system that supports automatic synchronization and verification of backup files across multiple systems.

System Architecture

• Server (db-sync-server): Receives and stores backup files, provides API interfaces
• Client (db-sync-client): Periodically checks and uploads backup files

Features

Security Features

• JWT token authentication
• MD5 file integrity check
• File size verification
• Secure HTTP transmission

Stability Features

• Automatic retry mechanism
• Error recovery
• Logging
• Health checks

Synchronization Strategies

• Monthly Upload: Automatically uploads all backup files on the 20th of each month
• Daily Verification: Verifies file integrity from the 20th to the 26th of each month
• Incremental Sync: Only uploads changed files

Quick Start

1. Configure the Server

Edit server.config.json:

{
  "listen_addr": "0.0.0.0:3030",
  "data": [
    {
      "system_name": "System Name",
      "system_ip": "192.168.1.100",
      "backup_dir": "./backups/System Name",
      "auth_token": "your-secret-token-here"
    }
  ]
}

2. Configure the Client

Edit client.config.json:

{
  "data": [
    {
      "system_name": "System Name",
      "system_ip": "192.168.1.100",
      "backup_server_url": "http://server-ip:3030",
      "auth_token": "your-secret-token-here",
      "local_backup_dir": "./local/backups"
    }
  ]
}

3. Build the Project

cargo build --release

4. Generate Authentication Tokens

Use the built-in CLI tool to generate a 512-bit authentication token for each system:

Generate a Token for a Single System

cargo run --bin gen-auth-token generate --system-ip 192.168.1.100 --system-name "HR System"

Example output:

=== DB-Sync Authentication Token Generation ===
System IP: 192.168.1.100
System Name: HR System
Token Length: 512 bits
Generated Token: 1154e194e0668a3a37ce63a28c6fe72e0200b937fb1ebb6bbf80345a8bee00c6a612fe10dac35b7781f89a13116c3cf81941c316529b0e89b2ee8ebbf3ac4846

Batch Generate Tokens

1. First, create a configuration file batch_config.json:

{
  "secret_key": "your_custom_secret_key",
  "systems": [
    {
      "system_ip": "192.168.1.100",
      "system_name": "HR System"
    },
    {
      "system_ip": "192.168.1.101",
      "system_name": "Asset Management System"
    }
  ]
}

2. Then run the batch generation command:

cargo run --bin gen-auth-token batch --config-file batch_config.json --format json

Validate Token Format

cargo run --bin gen-auth-token validate --token "your_token_here"

View All Available Commands

cargo run --bin gen-auth-token -- --help

Generate Example Configuration

cargo run --bin gen-auth-token example

5. Run the System

Start the Server

cargo run --bin db-sync-server

Start the Client

cargo run --bin db-sync-client

6. Test the System

Run the test script:

# Windows
test_system.bat

# Linux/Mac
bash test_system.sh

API Endpoints

Health Check

GET /health

File Upload

POST /api/upload
Headers:
- authorization: Bearer <token>

Query Parameters:
- system_ip: System IP
- filename: Filename
- blake3: File blake3 hash
- size: File size

Body: File binary data

blake3 Query

POST /api/blake3
Headers:
- authorization: Bearer <token>

Query Parameters:
- system_ip: System IP
- filename: Filename

File Structure

db-sync/
├── src/
│   ├── db-sync-server.rs    # Server code
│   └── db-sync-client.rs    # Client code
├── server.config.json       # Server configuration
├── client.config.json       # Client configuration
├── test_data/              # Test data
├── test_system.bat         # Windows test script
├── test_system.sh          # Linux/Mac test script
└── README.md               # Documentation

Security Recommendations

1. Token Management: Use strong passwords for auth_token and change them regularly.
2. Network Security: Use HTTPS in production environments.
3. Access Control: Restrict IP addresses for API access.
4. Backup Encryption: Encrypt sensitive backup files.

Monitoring and Maintenance

Viewing Logs

The system uses structured logging, and the log level can be controlled via environment variables:

export RUST_LOG=info
cargo run --bin db-sync-server

Health Check

Periodically check the service status:

curl http://server-ip:3030/health

File Verification

Manually verify file integrity:

# Calculate blake3 checksum for local file
blake3 backup_file.sql

# Query server file blake3 checksum
curl -X POST http://server-ip:3030/api/blake3 \
  -H "authorization: Bearer <token>" \
  -d "system_ip=192.168.1.100&filename=backup_file.sql"

Troubleshooting

Common Issues

1. Connection Failure: Check network connection and firewall settings.
2. Authentication Failure: Verify that the auth_token is configured correctly.
3. File Checksum Failure: Check file integrity during transmission.
4. Insufficient Disk Space: Regularly clean up old backup files.

Debug Mode

Enable verbose logging:

export RUST_LOG=debug
cargo run --bin db-sync-server

License

This project is licensed under the MIT License.

Crates.io Information

Package Name: db-sync Version: 0.1.0 Minimum Rust Version: 1.70+ License: MIT Keywords: backup, sync, database, file-transfer, security

Release Notes

This package is ready for publication on crates.io, providing an enterprise-grade database backup synchronization solution.

Features

• 🔐 Secure token authentication mechanism
• 📁 File integrity verification (blake3)
• ⏰ Scheduled task management
• 🔄 Automatic retry and error recovery
• 📊 Detailed logging and monitoring

Use Cases

• Enterprise database backups
• Multi-system file synchronization
• Scheduled data backup tasks
• Secure file transfer

Contributing

Issues and Pull Requests are welcome to improve this project.

Changelog

v0.1.0

• Initial release
• Implemented basic client-server architecture
• Support for file upload and blake3 verification
• Added scheduled task functionality
• Comprehensive error handling and logging