1 unstable release
0.0.9 | Sep 8, 2022 |
---|---|
0.0.8 |
|
#14 in #market
23 downloads per month
14KB
291 lines
Crawler Data Client
This is a cli tool to help automate the market data distribution process.
Setup
First make sure rust is installed on the system with rustup.
Then install the cli using:
cargo install crawler_data_client
In order to decompress the files, you also need to have a working installation of the zstd package.
Usage
crawler_data_client --address <ADDRESS> --username <USERNAME> --password <PASSWORD> --crawler-type <CRAWLER_TYPE> --exchange <EXCHANGE> --market <MARKET> --symbol <SYMBOL> --month <MONTH> --output <OUTPUT>
OPTIONS:
-h, --help (display help text)
-a, --address <ADDRESS> ---------- (ftp server address, eg 12.34.56.78:21. Usually the port is 21)
-u, --username <USERNAME> -------- (your ftp login username)
-p, --password <PASSWORD> -------- (your ftp login password)
-c, --crawler-type <CRAWLER_TYPE> (options: trades, l2_events, l2_topk, bbo, funding_rate, ticker (raw only))
-e, --exchange <EXCHANGE> -------- (options: binance, ftx, bybit)
-m, --market <MARKET> ------------ (options: spot, linear_swap, inverse_swap)
-s, --symbol <SYMBOL> ------------ (eg. BTCUSDT)
-o, --output <OUTPUT> ------------ (the local download destination folder, eg ~/Downloads/crawler-data)
--month <MONTH> -------------- (optional. the year-month in ISO, eg 2022-08. required if not using --latest-day or --latest-month)
--day <DAY> ------------------ (optional. the day of the month, eg. 07. if not provided, will download whole month. this needs to be ISO, so days less than 10 need a zero in front)
--keep-compressed ------------ (optional. if set, the data will be kept compressed as zst. default is to decompress the file for usage)
--raw ------------------------ (optional. download the raw data instead of parsed)
--latest-day ----------------- (optional. download the latest day, ie the previous utc day at time of download)
--latest-month --------------- (optional. download the latest whole month, ie the month that the previous utc day is in)
Dependencies
~12MB
~203K SLoC