22 releases
0.7.0-beta.5 | Jan 29, 2024 |
---|---|
0.7.0-beta.3 | Dec 1, 2023 |
0.7.0-beta.2 | Nov 26, 2023 |
0.7.0-alpha.1 | Jul 30, 2023 |
0.3.2 | Oct 29, 2021 |
#395 in Network programming
748 downloads per month
Used in 3 crates
145KB
3K
SLoC
BGPKIT Broker
BGPKIT Broker is an online data API service that allows users to search for publicly available BGP archive files by time, collector, project, or data type. The service indexes the archives in close to real-time (delay is less than 5 minutes). Currently, we are indexing BGP table dump and updates files from RIPE RIS and RouteViews.
This Rust library provides SDK access to the BGPKIT Broker API as well as a command-line tool to start a self-hosted broker instance. Current BGPKIT Broker API is available at https://api.bgpkit.com/docs.
BGPKIT Broker is used in production at Cloudflare Radar powering its routing page and projects like BGP hijack detection and route leak detection.
Broker Rust SDK
Usage
Add the following dependency line to your project's Cargo.toml
file:
bgpkit-broker = "0.7.0-beta.1"
Example
You can run the follow example with cargo run --example query
(source code).
use bgpkit_broker::{BgpkitBroker, BrokerItem};
pub fn main() {
let broker = BgpkitBroker::new()
.ts_start("1634693400")
.ts_end("1634693400");
// method 1: create iterator from reference (so that you can reuse the broker object)
// same as `&broker.into_iter()`
for item in &broker {
println!("{}", item);
}
// method 2: create iterator from the broker object (taking ownership)
let items = broker.into_iter().collect::<Vec<BrokerItem>>();
assert_eq!(items.len(), 106);
}
bgpkit-broker
CLI Tool
bgpkit-broker
is a command-line application that packages many functionalities to allow users to self-host a BGPKIT Broker instance with ease.
Install
Install with cargo install bgpkit-broker@0.7.0-beta.5 --features cli
or check out the main branch and run cargo install --path . --features cli
.
If you are on a macOS environment, you can also use homebrew to install the pre-compiled binary (universal):
brew install bgpkit/tap/bgpkit-broker
Usage
bgpkit-broker
has the following subcommands:
A library and command-line to provide indexing and searching functionalities for public BGP data archive files over time.
Usage: bgpkit-broker [OPTIONS] <COMMAND>
Commands:
serve Serve the Broker content via RESTful API
update Update the Broker database
bootstrap Bootstrap the broker database
backup Backup Broker database
search Search MRT files in Broker db
latest Display latest MRT files indexed
help Print this message or the help of the given subcommand(s)
Options:
--no-log disable logging
--env <ENV>
-h, --help Print help
-V, --version Print version
serve
bgpkit-broker serve
is the main command to start the BGPKIT Broker service. It will start a web server that serves the API endpoints. It will also periodically update the local database unless the --no-update
flag is set.
Serve the Broker content via RESTful API
Usage: bgpkit-broker serve [OPTIONS]
Options:
-i, --update-interval <UPDATE_INTERVAL> update interval in seconds [default: 300]
--no-log disable logging
--bootstrap-parquet bootstrap from parquet file instead of DuckDB file
-h, --host <HOST> host address [default: 0.0.0.0]
-p, --port <PORT> port number [default: 40064]
-r, --root <ROOT> root path, useful for configuring docs UI [default: /]
--no-update disable updater service
--no-api disable API service
--full-bootstrap do a full database bootstrap from duckdb or parquet file
-h, --help Print help
-V, --version Print version
update
bgpkit-broker update
triggers a local database update manually. This command cannot be run at the same time as serve
because the active API will lock the database file.
Update the Broker database
Usage: bgpkit-broker update [OPTIONS]
Options:
--no-log disable logging
--bootstrap-parquet bootstrap from parquet file instead of DuckDB file
-h, --help Print help
-V, --version Print version
backup
bgpkit-broker update
runs a database backup and export the database to a duckdb file and a parquet file. This can be run while serve
is running.
Export broker database to parquet file
Usage: bgpkit-broker backup [OPTIONS]
Options:
--no-log disable logging
--bootstrap-parquet bootstrap from parquet file instead of DuckDB file
-h, --help Print help
-V, --version Print version
search
bgpkit-broker search
queries for MRT files using the default production API unless specified otherwise.
Search MRT files in Broker db
Usage: bgpkit-broker search [OPTIONS]
Options:
--no-log disable logging
-t, --ts-start <TS_START> Start timestamp
--bootstrap-parquet bootstrap from parquet file instead of DuckDB file
-T, --ts-end <TS_END> End timestamp
-p, --project <PROJECT> filter by route collector projects, i.e. `route-views` or `riperis`
-c, --collector-id <COLLECTOR_ID> filter by collector IDs, e.g. 'rrc00', 'route-views2. use comma to separate multiple collectors
-d, --data-type <DATA_TYPE> filter by data types, i.e. 'update', 'rib'
--page <PAGE> page number
--page-size <PAGE_SIZE> page size
-u, --url <URL>
-j, --json print out search results in JSON format instead of Markdown table
-h, --help Print help
-V, --version Print version
latest
bgpkit-broker latest
queries for the latest MRT files of each route collector from RouteViews and RIPE RIS.
- use
--collector COLLECTOR
to narrow down the display of the collector. - use
--outdated
flag to toggle showing only the files from collectors that have not been generating data timely - use
--json
flag to output to a JSON file instead of a Markdown table
Display latest MRT files indexed
Usage: bgpkit-broker latest [OPTIONS]
Options:
-c, --collector <COLLECTOR> filter by collector ID
--no-log disable logging
--env <ENV>
-u, --url <URL> Specify broker endpoint
-o, --outdated Showing only latest items that are outdated
-j, --json Print out search results in JSON format instead of Markdown table
-h, --help Print help
-V, --version Print version
Data Provider
If you have publicly available data and want to be indexed BGPKIT Broker service, please send us an email at data@bgpkit.com. Our back-end service is designed to be flexible and should be able to adapt to most data archiving approaches.
Dependencies
~5–28MB
~434K SLoC