#sql #sqlite #apalis

apalis-sql

SQL Storage for apalis. Use sqlite, postgres and mysql for background job processing

14 releases

new 0.4.1 Jun 2, 2023
0.4.0-alpha.5 Apr 9, 2023
0.4.0-alpha.3 Mar 24, 2023
0.3.5 Dec 23, 2022
0.2.0 May 14, 2021

#199 in Database interfaces

Download history 27/week @ 2023-02-10 56/week @ 2023-02-17 1/week @ 2023-02-24 9/week @ 2023-03-03 2/week @ 2023-03-10 19/week @ 2023-03-17 29/week @ 2023-03-24 25/week @ 2023-03-31 73/week @ 2023-04-07 189/week @ 2023-04-14 174/week @ 2023-04-21 84/week @ 2023-04-28 124/week @ 2023-05-05 153/week @ 2023-05-12 213/week @ 2023-05-19 241/week @ 2023-05-26

744 downloads per month
Used in apalis

MIT license

115KB
2.5K SLoC

Rust 2K SLoC // 0.0% comments SQL 285 SLoC // 0.0% comments

apalis

Simple, extensible multithreaded background job and messages processing library for Rust


Features

  • Simple and predictable job handling model.
  • Jobs handlers with a macro free API.
  • Take full advantage of the tower ecosystem of middleware, services, and utilities.
  • Runtime agnostic - Use tokio, smol etc.
  • Optional Web interface to help you manage your jobs.

apalis job processing is powered by tower::Service which means you have access to the tower middleware.

apalis has support for:

Source Crate Example
Cron Jobs
Redis
Sqlite
Postgres
MySQL
Amqp
From Scratch

Getting Started

To get started, just add to Cargo.toml

[dependencies]
apalis = { version = "0.4", features = ["redis"] }

Usage

use apalis::prelude::*;
use apalis::redis::RedisStorage;
use serde::{Deserialize, Serialize};
use anyhow::Result;

#[derive(Debug, Deserialize, Serialize)]
struct Email {
    to: String,
}

impl Job for Email {
    const NAME: &'static str = "apalis::Email";
}

async fn email_service(job: Email, ctx: JobContext) {
    info!("Do something");
}

#[tokio::main]
async fn main() -> Result<()> {
    std::env::set_var("RUST_LOG", "debug");
    env_logger::init();
    let redis = std::env::var("REDIS_URL").expect("Missing env variable REDIS_URL");
    let storage = RedisStorage::new(redis).await?;
    Monitor::new()
        .register_with_count(2, move |index| {
            WorkerBuilder::new(format!("email-worker-{index}"))
                .with_storage(storage.clone())
                .build_fn(email_service)
        })
        .run()
        .await
}

Then

//This can be in another part of the program or another application eg a http server
async fn produce_route_jobs(storage: &RedisStorage<Email>) -> Result<()> {
    let mut storage = storage.clone();
    storage
        .push(Email {
            to: "test@example.com".to_string(),
        })
        .await?;
}

Feature flags

  • tracing (enabled by default) — Support Tracing 👀
  • redis — Include redis storage
  • postgres — Include Postgres storage
  • sqlite — Include SQlite storage
  • mysql — Include MySql storage
  • cron — Include cron job processing
  • sentry — Support for Sentry exception and performance monitoring
  • prometheus — Support Prometheus metrics
  • retry — Support direct retrying jobs
  • timeout — Support timeouts on jobs
  • limit — 💪 Limit the amount of jobs
  • filter — Support filtering jobs based on a predicate
  • extensions — Add a global extensions to jobs

Storage Comparison

Since we provide a few storage solutions, here is a table comparing them:

Feature Redis Sqlite Postgres Sled Mysql Mongo Cron
Scheduled jobs x x
Retry jobs x x
Persistence x x BYO
Rerun Dead jobs x x x

How apalis works

sequenceDiagram
    participant App
    participant Worker
    participant Producer

    App->>+Producer: Add job to queue
    Producer-->>+Worker: Job data
    Worker->>+Producer: Update job status to 'running' via Layer
    Producer-->>-Worker: Confirmation
    Worker->>+App: Notify job started via Layer
    loop job execution
        Worker-->>-App: Report job progress via Layer
    end
    Worker->>+Producer: Update job status to 'completed' via Layer

Web UI

If you are running apalis Board, you can easily manage your jobs. See a working rest API example here

Thanks to

  • tower - Tower is a library of modular and reusable components for building robust networking clients and servers.
  • redis-rs - Redis library for rust
  • sqlx - The Rust SQL Toolkit

Roadmap

v 1.0

  • Refactor the crates structure
  • Mocking utilities
  • Support for SurrealDB and Mongo
  • Lock free for Postgres
  • Add more utility layers
  • Use extractors in job fn structure
  • Polish up documentation
  • Improve and standardize apalis Board
  • Benchmarks

v 0.4

  • Move from actor based to layer based processing
  • Graceful Shutdown
  • Allow other types of executors apart from Tokio
  • Mock/Test Worker
  • Improve monitoring
  • Add job progress via layer
  • Add more sources

v 0.3

  • Standardize API (Storage, Worker, Data, Middleware, Context )
  • Introduce SQL
  • Implement layers for Sentry and Tracing.
  • Improve documentation
  • Organized modules and features.
  • Basic Web API Interface
  • Sql Examples
  • Sqlx migrations

v 0.2

  • Redis Example
  • Actix Web Example

Resources

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Dependencies

~11–49MB
~874K SLoC