#drain #log #grok #categorization

drain-rs

Automatically cluster semi-formatted text with the drain algorithm

3 releases (breaking)

0.3.0 May 31, 2021
0.2.0 Sep 22, 2020
0.1.0 Sep 22, 2020

#962 in Algorithms

Custom license

30KB
615 lines

drain-rs

Drain provides a mechanism for online log categorization.

This version provides:

  • serialization/deserialization of drain state via serde json
  • support for GROK patterns for more accurate categories and variable filtering

The goal of this particular project is to provide a nice, fast, rust upgrade to the original drain implementation. Original paper here:

This is a WIP, 0.2.x

Installing

[dependencies]
drain-rs = "0.2.0"

Using drain for clustering

To use drain for clustering:

//Create new drain tree object
let mut drain = DrainTree::new()
// Add log lines and see their group:
let log_group = drain.add_log_line(s.as_str());

To use drain with grok:

let mut g = grok::Grok::with_patterns();
let filter_patterns = vec![
    "blk_(|-)[0-9]+",     //blockid
    "%{IPV4:ip_address}", //IP
    "%{NUMBER:number}",   //Num
];
// Build new drain tree
let mut drain = DrainTree::new()
    .filter_patterns(filter_patterns)
    .max_depth(4)
    .max_children(100)
    .min_similarity(0.5)
    // HDFS log pattern, variable format printout in the content section
    .log_pattern("%{NUMBER:date} %{NUMBER:time} %{NUMBER:proc} %{LOGLEVEL:level} %{DATA:component}: %{GREEDYDATA:content}", "content")
    // Compile all the grok patterns so that they can be used
    .build_patterns(&mut g);

Dependencies

~4MB
~120K SLoC