#log-file #logging #async #metrics #tracing #data-channel

fast_log

Rust async log High-performance asynchronous logging

15 stable releases

1.6.16 Mar 9, 2024
1.6.12 Dec 28, 2023
1.6.10 Oct 31, 2023
1.5.57 Jul 20, 2023
1.0.3 Mar 8, 2020

#25 in Debugging

Download history 527/week @ 2023-11-25 401/week @ 2023-12-02 428/week @ 2023-12-09 436/week @ 2023-12-16 412/week @ 2023-12-23 364/week @ 2023-12-30 678/week @ 2024-01-06 688/week @ 2024-01-13 459/week @ 2024-01-20 581/week @ 2024-01-27 262/week @ 2024-02-03 282/week @ 2024-02-10 756/week @ 2024-02-17 1096/week @ 2024-02-24 854/week @ 2024-03-02 703/week @ 2024-03-09

3,480 downloads per month
Used in 5 crates

MIT license

62KB
1.5K SLoC

fast_log

Build Status GitHub release

the fast log . This crate uses #! [forbid(unsafe_code)] to ensure everything is implemented in 100% Safe Rust.

A log implementation for extreme speed, using Crossbeam/channel ,once Batch write logs,fast log date, Appender architecture, appender per thread

  • Low overhead, log write based on thread, also support tokio/Future

  • High performance, use lockless message queue, log is stored in queue, then flush disk. It does not block the caller

  • Full APPEND mode file writing, high efficiency for solid state/mechanical disk (solid state and mechanical disk sequential write performance is better than random write)

  • When channel pressure increases, logs can be written in batches at a time

  • Built-in ZIP compression, compressed file name date + serial number, no need to worry about the log file is too large

  • Built-in log segmentation, custom log full number of immediately split

  • Built-in filtering configuration support, can be customized to filter out other library printed logs

  • Support custom compression algorithms, such as ZIP and LZ4

  • Support use log::logger().flush() method wait to flush disk

  • Simple and efficient Appender architecture.Both configuration and customization are simple

  • Support custom file type, just like mmap/file


              -----------------
log data->    | main channel(crossbeam)  |   ->          
              ----------------- 
                                        ----------------                                    ----------------------
                                  ->    |thread channel)|  -> background thread  |    appender1  |
                                        ----------------                                    ----------------------

                                        ----------------                                    ----------------------
                                  ->    |thread channel)|  -> background thread  |    appender2  |
                                        ----------------                                    ----------------------

                                        ----------------                                    ----------------------
                                  ->    |thread channel)|  -> background thread  |    appender3  |
                                        ----------------                                    ----------------------

                                        ----------------                                    ----------------------
                                  ->    |thread channel)|  -> background thread  |    appender4  |
                                        ----------------                                    ----------------------


  • How fast is?

  • no flush(chan_len=1000000) benches/log.rs

//MACOS(Apple M1MAX-32GB)
test bench_log ... bench:          85 ns/iter (+/- 1,800)
  • all log flush into file(chan_len=1000000) example/bench_test_file.rs
//MACOS(Apple M1MAX-32GB)
test bench_log ... bench:          323 ns/iter (+/- 0)
  • how to use?
log = "0.4"
fast_log = {version = "1.5"}

or enable zip/lz4/gzip Compression library

log = "0.4"
# "lz4","zip","gzip"
fast_log = {version = "1.5" , features = ["lz4","zip","gzip"]}

Performance optimization(important)

  • use chan_len(Some(100000)) Preallocating channel memory reduces the overhead of memory allocation,for example:
use log::{error, info, warn};
fn  main(){
    fast_log::init(Config::new().file("target/test.log").chan_len(Some(100000))).unwrap();
    log::info!("Commencing yak shaving{}", 0);
}

Use Log(Console)

use log::{error, info, warn};
fn  main(){
    fast_log::init(Config::new().console().chan_len(Some(100000))).unwrap();
    log::info!("Commencing yak shaving{}", 0);
}

Use Log(Console Print)

use log::{error, info, warn};
fn  main(){
    fast_log::init(Config::new().console().chan_len(Some(100000))).unwrap();
    fast_log::print("Commencing print\n".into());
}

Use Log(File)

use fast_log::{init_log};
use log::{error, info, warn};
fn  main(){
    fast_log::init(Config::new().file("target/test.log").chan_len(Some(100000))).unwrap();
    log::info!("Commencing yak shaving{}", 0);
    info!("Commencing yak shaving");
}

Split Log(.log packer)

use fast_log::plugin::file_split::RollingType;
use fast_log::consts::LogSize;
use fast_log::plugin::packer::LogPacker;

#[test]
pub fn test_file_compation() {
    fast_log::init(Config::new()
        .console()
        .chan_len(Some(100000))
        .file_split("target/logs/",
                    LogSize::MB(1),
                    RollingType::All,
                    LogPacker{})).unwrap();
    for _ in 0..200000 {
        info!("Commencing yak shaving");
    }
    log::logger().flush();
}

Split Log(mmap)

  • Mmap (memory map) maps files or devices into memory and enables direct memory access.
  • It is supported on Windows, Linux and macOS to map a file's contents into a program's address space.
  • This allows very fast I/O without intermediate copying. Mmap is efficient for reading and writing files as it utilizes caching.
  • It also enables easy sharing of memory between processes. Overall mmap provides fast, convenient file access while the operating system handles low level details like caching and swapping.
use fast_log::config::Config;
use fast_log::consts::LogSize;
use fast_log::plugin::file_mmap::MmapFile;
use fast_log::plugin::file_split::RollingType;
use fast_log::plugin::packer::LogPacker;

fn main() {
    fast_log::init(
        Config::new()
            .chan_len(Some(100000))
            .console()
            .split::<MmapFile, LogPacker>(
                "target/logs/temp.log",
                LogSize::MB(1),
                RollingType::All,
                LogPacker {},
            ),
    )
    .unwrap();
    for _ in 0..40000 {
        log::info!("Commencing yak shaving");
    }
    log::logger().flush();
    println!("you can see log files in path: {}", "target/logs/");
}

Custom Log(impl do_log method)
use fast_log::{LogAppender};
use log::{error, info, warn};

pub struct CustomLog{}
impl LogAppender for CustomLog{
    fn do_log(&mut self, record: &FastLogRecord) {
        print!("{}",record);
    }
}
fn  main(){
    let wait = fast_log::init(Config::new().custom(CustomLog {}).chan_len(Some(100000))).unwrap();
    info!("Commencing yak shaving");
    log::logger().flush();
}

Dependencies

~2–11MB
~104K SLoC