14 releases

new 0.2.3 Dec 11, 2024
0.2.2 Nov 29, 2024
0.2.0 Oct 3, 2024
0.1.5 Jul 19, 2024
0.1.0 Mar 15, 2024

#476 in Database interfaces

Download history 142/week @ 2024-08-19 5/week @ 2024-08-26 147/week @ 2024-09-02 218/week @ 2024-09-16 34/week @ 2024-09-23 164/week @ 2024-09-30 13/week @ 2024-10-07 4/week @ 2024-10-14 142/week @ 2024-11-18 116/week @ 2024-11-25 18/week @ 2024-12-02

276 downloads per month

Apache-2.0

515KB
13K SLoC

Micromegas

Micromegas is a unified and scalable observability stack. It can help you collect and query logs, metrics and traces.


lib.rs:

Micromegas is a unified and scalable observability stack. It can help you collect and query logs, metrics and traces.

Very high level architecture

┌─────────────────┐       
│ rust application│──────▶
└─────────────────┘       ┌─────────┐     ┌───────┐     ┌─────────┐     ┌──────────┐
                          │ingestion│────▶│pg & S3│◀────│analytics│◀────│python API│
┌─────────────────┐       └─────────┘     └───────┘     └─────────┘     └──────────┘
│ unreal engine   │──────▶
└─────────────────┘      

Rust Instrumentation

For rust applications, use micromegas::tracing for minimal overhead. Interoperability with tokio tracing's logs is also enabled by default.

Unreal instrumentation

MicromegasTracing should be added to Unreal's Core module and MicromegasTelemetrySink can be added to a game or to a high level plugin. See https://github.com/madesroches/micromegas/tree/main/unreal for implementation. It has been tested in editor, client and server builds on multiple platforms.

Telemetry ingestion server

https://github.com/madesroches/micromegas/blob/main/rust/telemetry-ingestion-srv/src/main.rs

Analytics server

https://github.com/madesroches/micromegas/blob/main/rust/analytics-srv/src/main.rs

Lakehouse daemon

https://github.com/madesroches/micromegas/blob/main/rust/telemetry-admin-cli/src/telemetry_admin.rs (with crond argument)

Python API

https://pypi.org/project/micromegas/

Local developer configuration

For testing purposes, you can run the entire stack on your local workstation.

Environment variables

  • MICROMEGAS_DB_USERNAME and MICROMEGAS_DB_PASSWD: used by the database configuration script
  • export MICROMEGAS_TELEMETRY_URL=http://localhost:9000
  • export MICROMEGAS_SQL_CONNECTION_STRING=postgres://{uname}:{passwd}@localhost:5432
  • export MICROMEGAS_OBJECT_STORE_URI=file:///some/local/path
  1. Clone the github repository
> git clone https://github.com/madesroches/micromegas.git
  1. Start a local instance of postgresql (requires docker and python)
> cd micromegas/local_test_env/db
> ./run.py
  1. In a new shell, start the ingestion server
> cd micromegas/rust
> cargo run -p telemetry-ingestion-srv -- --listen-endpoint-http 127.0.0.1:9000
  1. In a new shell, start the analytics server
> cd micromegas/rust
> cargo run -p analytics-srv
  1. In a new shell, start the daemon
> cd micromegas/rust
> cargo run -p telemetry-admin -- crond
  1. In a python interpreter, query the analytics service
# local connection test
import datetime
import micromegas
BASE_URL = "http://localhost:8082/"
client = micromegas.client.Client(BASE_URL)
now = datetime.datetime.now(datetime.timezone.utc)
begin = now - datetime.timedelta(days=1)
end = now
sql = """
SELECT *
FROM log_entries
ORDER BY time DESC
LIMIT 10
;"""
client.query_view("log_entries", "global", begin, end, sql)

Dependencies

~107MB
~2M SLoC