chapaty v0.1.0

Chapaty is the open source software to backtast trading strategies on different markets. Your trading results on chapaty-ai are computed by this library. To use this library you need to provide your own data and follow the conventions for the historical data below.

Data Providers and Markets

Currently the following data providers and markets are supported. The historical market data reaches back until 2006 for some markets.

Profit and Loss Report

When Backtesting your bot, chapaty will generate a profit and loss report of the following form.

Performance Report

Additionally to the profit and loss report, chapaty generates a performance report of the following form.

Trade Breakdown Report

Opening Data in Excel

The reports are simple .csv files, where:

• Columns are separated with a comma ,
• Floating point numbers use a decimal point .

If you want to open a report in Excel and you don't get a nice formatted file, try the following steps:

1. Open the .csv file in a simple text editior
2. Replace all , with ;
3. Replace all . with ,
4. Save the file
5. Reopen the file in Excel agein

Directory Layout

All the data is stored inside a Google Cloud Storage bucket. There are two buckets. We use one bucket to store the historical market data. We have a second bucket to cache the computed results to decrease the computation time to evaluate a bot on historical market data.

Historical Market Data

We use the following directory layout. It is for illustrative purposes, as Google Cloud Storage does not have folders or subdirectories. The data is stored as a .csv inside the "bucket"/{provider}/{aggTrades | ohlc-{ts} | ...} directory. The files follow a strict naming convention.

.
├── {provider}              # binance / cme / ...
    ├── aggTrades/         
    │   ├── {market}-aggTrades-{year}[-{month?}-{day?}].csv
    │   ├── btcusdt-aggTrades-2023-01.csv
    │   ├── ...        
    ├── ohlc/               # For ts in { 1w | 1d | 12h | 8h | 6h | 4h | 3h | 2h | 1h | 30m | 15m | 5m | 1m | 30s | 15s | 1s }
    │   ├── {market}-{ts}-{year}[-{month?}-{day?}].csv
    │   ├── btcusdt-1m-2023-01.csv
    │   ├── ...
    ├── ohlcv/              # For ts in { 1w | 1d | 12h | 8h | 6h | 4h | 3h | 2h | 1h | 30m | 15m | 5m | 1m | 30s | 15s | 1s }
    │   ├── {market}-{ts}-{year}[-{month?}-{day?}].csv
    │   ├── btcusdt-1m-2023-01.csv
    │   ├── ...
    ├── tick/               
        ├── {market}-tick-{year}[-{month?}-{day?}].csv
        ├── btcusdt-tick-2023-01.csv
        ├── ...

Cached Data

The cached data is stored in a separate bucket, following this directory layout.

.
├── {strategy}                              # ppp / magneto / postNews / ...
    ├── {market}                            # 6e / btcusdt / ...
        ├── {year}                          # 2023 / 2022 / ...
            ├── {time_interval}             # none / TimeFrame::to_string()
                ├── {time_frame}            # { 1w | 1d | 12h | 8h | 6h | 4h | 3h | 2h | 1h | 30m | 15m | 5m }
                    ├── aggTrades.csv       # Output of processed `.csv` files
                    ├── ohlc-{ts}.csv       # Output of processed `.csv` files
                    ├── ohlcv-{ts}.csv      # Output of processed `.csv` files
                    ├── tick.csv            # Output of processed `.csv` files
                    ├── tpo-{ts}.csv        # Output of processed `.csv` files
                    ├── vol-tick.csv        # Contains volume profile, which is computed by tick data
                    ├── vol-aggTrades.csv   # Contains volume profile, which is computed by aggTrades data
                    ├── ...

Test Data

We maintain an additional bucket for test data. The directory layout is identical to the above ones, with one special directory ohter.

.
├── {...directory layout}
│
├── other                   # Special only directory
    ├── test_file1.csv      # Test files for unit tests
    ├── test_file2.csv      # ...
    ├── ...
    ├── gcp/                # Contains test files for the `gcp` module
    ├── market_profile/     # Contains test files for the `market_profile` module
    ├── ppp/                # Contains test files for the `ppp` module
    └── ...                 # etc.

Downloading from GCP

To download multiple files from the Google Cloud Platform do the following steps.

1. Open the gsutil command line tool
2. Paste the command e.g. gsutil cp -r gs://bucket/folder .
3. Run $ zip -r folder.zip folder/ to zip your files
4. In the gsutil commoand line tool menu click the botton with the three dots on the top right corner
5. Select Download
6. Append the file folder.zip name you want to download to the file path

Renaming files

Use the rename command:

1. To lowercase: rename -f 'y/A-Z/a-z/' *.csv
2. Replace _ with -: rename 's/_/-/g' *.csv
3. Replace 6btc-1m-01-01-2021-31-12-2021.csv to 6btc-1m-2021-31-12-2021.csv by rename 's/^(\w+-\w+-)(\d{2}-\d{2}-)(\d{4}-\d{2}-\d{2}-\d{4})\.csv$/$1$3.csv/' *.csv and then 6btc-1m-2021-31-12-2021.csv to 6btc-1m-2021.csv by rename 's/^(.{12}).*\.csv$/$1.csv/' *.csv (replace 12 in (.{12}) with 10 if you want to keep first ten letters)