14 releases
new 0.2.3 | Dec 11, 2024 |
---|---|
0.2.2 | Nov 29, 2024 |
0.2.0 | Oct 3, 2024 |
0.1.5 | Jul 19, 2024 |
0.1.0 | Mar 15, 2024 |
#476 in Database interfaces
276 downloads per month
515KB
13K
SLoC
Micromegas
Micromegas is a unified and scalable observability stack. It can help you collect and query logs, metrics and traces.
lib.rs
:
Micromegas is a unified and scalable observability stack. It can help you collect and query logs, metrics and traces.
Very high level architecture
┌─────────────────┐
│ rust application│──────▶
└─────────────────┘ ┌─────────┐ ┌───────┐ ┌─────────┐ ┌──────────┐
│ingestion│────▶│pg & S3│◀────│analytics│◀────│python API│
┌─────────────────┐ └─────────┘ └───────┘ └─────────┘ └──────────┘
│ unreal engine │──────▶
└─────────────────┘
Rust Instrumentation
For rust applications, use micromegas::tracing for minimal overhead. Interoperability with tokio tracing's logs is also enabled by default.
Unreal instrumentation
MicromegasTracing
should be added to Unreal's Core module and MicromegasTelemetrySink
can be added to a game or to a high level plugin. See https://github.com/madesroches/micromegas/tree/main/unreal for implementation. It has been tested in editor, client and server builds on multiple platforms.
Telemetry ingestion server
https://github.com/madesroches/micromegas/blob/main/rust/telemetry-ingestion-srv/src/main.rs
Analytics server
https://github.com/madesroches/micromegas/blob/main/rust/analytics-srv/src/main.rs
Lakehouse daemon
https://github.com/madesroches/micromegas/blob/main/rust/telemetry-admin-cli/src/telemetry_admin.rs (with crond
argument)
Python API
https://pypi.org/project/micromegas/
Local developer configuration
For testing purposes, you can run the entire stack on your local workstation.
Environment variables
MICROMEGAS_DB_USERNAME
andMICROMEGAS_DB_PASSWD
: used by the database configuration scriptexport MICROMEGAS_TELEMETRY_URL=http://localhost:9000
export MICROMEGAS_SQL_CONNECTION_STRING=postgres://{uname}:{passwd}@localhost:5432
export MICROMEGAS_OBJECT_STORE_URI=file:///some/local/path
- Clone the github repository
> git clone https://github.com/madesroches/micromegas.git
- Start a local instance of postgresql (requires docker and python)
> cd micromegas/local_test_env/db
> ./run.py
- In a new shell, start the ingestion server
> cd micromegas/rust
> cargo run -p telemetry-ingestion-srv -- --listen-endpoint-http 127.0.0.1:9000
- In a new shell, start the analytics server
> cd micromegas/rust
> cargo run -p analytics-srv
- In a new shell, start the daemon
> cd micromegas/rust
> cargo run -p telemetry-admin -- crond
- In a python interpreter, query the analytics service
# local connection test
import datetime
import micromegas
BASE_URL = "http://localhost:8082/"
client = micromegas.client.Client(BASE_URL)
now = datetime.datetime.now(datetime.timezone.utc)
begin = now - datetime.timedelta(days=1)
end = now
sql = """
SELECT *
FROM log_entries
ORDER BY time DESC
LIMIT 10
;"""
client.query_view("log_entries", "global", begin, end, sql)
Dependencies
~107MB
~2M SLoC