6 releases
0.1.7 | Jun 1, 2019 |
---|---|
0.1.6 | Mar 26, 2019 |
0.1.4 | Feb 5, 2019 |
#83 in #downloader
54KB
1.5K
SLoC
Pikmin
An extensible downloader for obtaining trade data simultaneously from exchanges' API.
pikmin
is a trade (execution) data downloader for crypto-currency exchanges,
such as BitMex, bitFlyer, Liquid, etc. This library provides not only some pre-composed
downloaders, but also ability to build a custom downloader for users' demand.
Pre-composed downloaders
Currently, this library has the following downloaders:
- BitMex (specify time, chronologically)
- bitFlyer (specify id, reverse-chronologically)
- Liquid (specify time, chronologically)
Built-in Writer
Writer
is a processor between trade data and destination (typically DB).
Pikmin has some built-in writers:
- MySQL
- stdout
You can create your own writer easily.
Example
A simple downloader for Liquid with writing to stdout.
This program creates ./qn-progress.txt
for recording progress,
so delete it if you want to run again from the starting point.
use std::path::PathBuf;
use std::thread;
use std::time::Duration;
use chrono::offset::TimeZone;
use chrono::Utc;
use pikmin::downloader::Downloader;
use pikmin::FileRecorder;
use pikmin::LiquidDownloader;
use pikmin::StdOutWriter;
fn main() {
// by using thread, you can run multiple downloaders
let liquid = thread::spawn(move || {
while {
// download data from 2019-01-01T01:01:01Z to 2019-01-01T01:03:01Z
// output the downloaded data to standard out (println!)
let downloader = LiquidDownloader::new(
Utc.ymd(2019, 1, 1).and_hms(1, 1, 1),
Utc.ymd(2019, 1, 1).and_hms(1, 3, 1),
);
// Locate progress file to `/tmp/qn-progress.txt`.
// You can control the starting point of downloading
// by preparing this file before you run.
let mut recorder = FileRecorder::new(PathBuf::from("/tmp/qn-progress.txt"));
// write out to standard out. simple writer for debugging
let mut writer = StdOutWriter::default();
println!("start QnDownloader");
// run!
match downloader.run(&mut writer, &mut recorder) {
Ok(_) => {
println!("finished");
false
}
Err(e) => {
println!("error: {}", e);
println!("retry...");
thread::sleep(Duration::from_secs(5));
true
}
}
} {}
});
match liquid
.join()
{
Ok(_) => println!("finish all"),
Err(_) => println!("threading error"),
}
}
Other examples can be found in ./examples
.
Future work
- create pre-composed downloaders for other exchanges
- parameterize the direction of downloading (chronologically or not)
- abstraction of the progress writer
Dependencies
~28MB
~592K SLoC