Multiline

multiline is a CLI utility that can be used to transform realtime log message streams (e.g. log entries from application log file) that contain messages spanning multiple lines (e.g. stack traces) into streams with one message per line.

It can be useful as pre-processing step before sending log file data to syslog which would otherwise interpret messages spanning over multiple lines as separate messages one per line.

Usage

multiline program will read UTF-8 data from from standard input, one line at a time and write messages spawning potentially multiple lines joined by --join string to standard output.

To configure message beginning or end matching algorithm --message-pattern regex, --negate and --match-last flags can be used. --strip-pattern flag can by used to remove matched message pattern from output.

Additionally multiline can handle multiple streams of log data interlaced together in standard input with use of --stream-id-pattern regex to match each stream identifying pattern.

Real time processing parameters can be configured with --max-size flag which limits maximum number of lines that each stream will accumulate and --max-duration which limits time duration that each message will continue to aggregate lines for. After one of this limits is reached concatenated message is flushed to the output.

Specifying --timestamp flag will enable printing of timestamps (in ISO 8601 UTC format) for each message constructed based on time when first line of the message was read.

Examples

Joining stack traces

Match every line that starts with space as belonging to previous message with negate.

multiline --message-pattern '^\s+' --negate --strip-pattern --join "; " <<EOF
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Hello world!
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
EOF

Produces:

Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Hello world!
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Joining lines ending with special escape

Use match-last with negate to anchor on the lines without the escape. Add strip-pattern to ensure that the escape is not part of the output.

multiline --message-pattern ' >$' --negate --match-last --strip-pattern --join "; " <<EOF
Exception in thread "main" java.lang.NullPointerException >
at com.example.myproject.Book.getTitle(Book.java:16) >
at com.example.myproject.Author.getBookTitles(Author.java:25) >
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException >
at com.example.myproject.Book.getTitle(Book.java:16) >
at com.example.myproject.Author.getBookTitles(Author.java:25) >
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Hello world!
Exception in thread "main" java.lang.NullPointerException >
at com.example.myproject.Book.getTitle(Book.java:16) >
at com.example.myproject.Author.getBookTitles(Author.java:25) >
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
EOF

Produces:

Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Hello world!
Exception in thread "main" java.lang.NullPointerException; at com.example.myproject.Book.getTitle(Book.java:16); at com.example.myproject.Author.getBookTitles(Author.java:25); at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Join messages from multiple interlaced streams of log data

The stream-id-pattern regex will match the stream name. For each stream a separate concatenation buffer will be maintained. Timestamp is used to match the beginning of the message in each stream.

multiline --message-pattern '^20' --stream-id-pattern '.*: ' --join " " <<EOF
bar: 2019-07-05 15:42:16 Lorem ipsum dolor sit amet,
foo: 2019-07-05 15:42:16 Lorem ipsum dolor sit amet,
foo: consectetur adipiscing elit
bar: consectetur adipiscing elit
foo: 2019-07-05 15:42:17 Phasellus eleifend scelerisque lorem,
foo: a placerat ex dictum iaculis.
foo: 2019-07-05 15:42:18 Nam porta hendrerit fermentum.
foo: 2019-07-05 15:42:19 Vivamus vitae faucibus purus.
bar: 2019-07-05 15:42:17 Phasellus eleifend scelerisque lorem,
foo: 2019-07-05 15:42:20 Cras suscipit lacus ex.
bar: a placerat ex dictum iaculis.
bar: 2019-07-05 15:42:18 Nam porta hendrerit fermentum.
bar: 2019-07-05 15:42:19 Vivamus vitae faucibus purus.
foo: 2019-07-05 15:42:21 Phasellus sed nulla faucibus,
foo: bibendum dolor vitae,
foo: faucibus dui.
bar: 2019-07-05 15:42:20 Cras suscipit lacus ex.
bar: 2019-07-05 15:42:21 Phasellus sed nulla faucibus,
bar: bibendum dolor vitae,
bar: faucibus dui.
EOF

Produces:

foo: 2019-07-05 15:42:16 Lorem ipsum dolor sit amet, consectetur adipiscing elit
foo: 2019-07-05 15:42:17 Phasellus eleifend scelerisque lorem, a placerat ex dictum iaculis.
foo: 2019-07-05 15:42:18 Nam porta hendrerit fermentum.
bar: 2019-07-05 15:42:16 Lorem ipsum dolor sit amet, consectetur adipiscing elit
foo: 2019-07-05 15:42:19 Vivamus vitae faucibus purus.
bar: 2019-07-05 15:42:17 Phasellus eleifend scelerisque lorem, a placerat ex dictum iaculis.
bar: 2019-07-05 15:42:18 Nam porta hendrerit fermentum.
foo: 2019-07-05 15:42:20 Cras suscipit lacus ex.
bar: 2019-07-05 15:42:19 Vivamus vitae faucibus purus.
bar: 2019-07-05 15:42:20 Cras suscipit lacus ex.
foo: 2019-07-05 15:42:21 Phasellus sed nulla faucibus, bibendum dolor vitae, faucibus dui.
bar: 2019-07-05 15:42:21 Phasellus sed nulla faucibus, bibendum dolor vitae, faucibus dui.

Timestamp messages

Enable timestamps with --timestamp flag.

seq 1 10 | while read I; do echo $I; sleep 0.2; done | multiline --message-pattern '.' --timestamp

Produces:

2019-07-12T13:29:12.872375Z 1
2019-07-12T13:29:13.071095Z 2
2019-07-12T13:29:13.273488Z 3
2019-07-12T13:29:13.475585Z 4
2019-07-12T13:29:13.677506Z 5
2019-07-12T13:29:13.879438Z 6
2019-07-12T13:29:14.082132Z 7
2019-07-12T13:29:14.284793Z 8
2019-07-12T13:29:14.487547Z 9
2019-07-12T13:29:14.690270Z 10