c8-backup

A CLI tool to automate backup and restore of Camunda Platform 8 deployments.

Current Status

⚠️ Experimental ⚠️

Features:

• List backups
• Create backups
• Restore backups
• Dry runs

Components:

• Zeebe
• Operate
• Tasklist
• Optimize

Deployments:

• Remote Camunda Platform 8 Helm installation (running locally, connecting through the current kubernetes context).
• Local Camunda Platform 8 Helm installation (running as a pod inside the cluster).

Usage

Warning This tool is experimental and not safe for use yet! Running it can result in complete dataloss.

Download a pre-built binary from the latest release or build from sources with

cargo install c8-backup

Currently, this tool is meant to run locally. It will connect to your current kubernetes context and tries to find C8 components there.

Listing backups

The list command lists recent backups by status and also shows which is the most recent, usable, backup. This means that the backup is successfully completed by all components.

$ c8-backup list
c8_backup::list::list{}
  c8_backup::list::Zeebe{}
    0ms  INFO c8_backup::list 2 backups Completed: 1683214620, 1683214072, ...
  
  c8_backup::list::Operate{}
    0ms  INFO c8_backup::list 2 backups Completed: 1683214620, 1683214072, ...
  
  1060ms  INFO c8_backup::list The most recent usable backup is 1683214620
  1060ms  INFO c8_backup::list This backup was created 8 minutes ago at 2023-05-04 15:37:00 UTC

Creating backups

The create command starts without any additional confirmation and will take the following steps:

1. Take a backup of Operate
2. Pause Zeebe exporting
3. Take a backup of exported Zeebe records
4. Take a backup of Zeebe
5. Resume Zeebe exporting

Resuming exporting is crucial and this command tries to resume exporting if any error occurs while taking a backup but you should manually confirm that exporting resumed, for example by following the log output.

$ c8-backup create
c8_backup::create::create{}
  c8_backup::create::try_backup{backup_id=1683214620}
    c8_backup::create::backup_operate{backup_id=1683214620}
      c8_backup::operate::take_backup{backup_id=1683214620}
      
      267ms  INFO c8_backup::create Started backup
      521ms  INFO c8_backup::create Checking again in 5 seconds, state is InProgress
      5870ms  INFO c8_backup::create Checking again in 5 seconds, state is InProgress
      11233ms  INFO c8_backup::create Checking again in 5 seconds, state is Incomplete
      16598ms  INFO c8_backup::create Backup completed
    
    c8_backup::zeebe::pause_exporting{}
    
    c8_backup::create::backup_zeebe_export{backup_id=1683214620}
      c8_backup::elasticsearch::take_snapshot{req=SnapshotRequest { indices: "zeebe-record*", feature_states: ["none"] }, name="camunda_zeebe_records_1683214620"}
      
    
    c8_backup::create::backup_zeebe{backup_id=1683214620}
      c8_backup::zeebe::take_backup{backup_id=1683214620}
      
      280ms  INFO c8_backup::create Started backup
      831ms  INFO c8_backup::create Checking again in 5 seconds, state is InProgress
      6180ms  INFO c8_backup::create Backup completed
    
    c8_backup::zeebe::resume_exporting{}```

### Restoring backups

The `restore` commands starts without any additional confirmation and will take the following steps:
1. Find the latest backup from Zeebe and Operate
2. Stop Zeebe and Operate
3. Delete **all** indices from Elasticsearch
4. Restore Elasticsearch indices based on backups
5. Delete all Zeebe data
6. Restore Zeebe data based on backup
7. Start Zeebe and Operate

```shell
$ c8-backup restore
c8_backup::restore::restore{}
  c8_backup::restore::find_newest_backup{}
    1573ms  INFO c8_backup::restore Using backup 1683214620
  
  c8_backup::restore::shutdown_apps{}
    331ms  INFO c8_backup::restore Shut down ccs23-dev-zeebe-gateway
    380ms  INFO c8_backup::restore Shut down ccs23-dev-operate
    431ms  INFO c8_backup::restore Shut down ccs23-dev-zeebe
  
  c8_backup::restore::delete_indices{}
    c8_backup::elasticsearch::get_all_indices{}
    
    434ms  INFO c8_backup::elasticsearch Deleted index operate-flownode-instance-8.2.0_
    695ms  INFO c8_backup::elasticsearch Deleted index operate-decision-instance-8.2.0_
    966ms  INFO c8_backup::elasticsearch Deleted index operate-list-view-8.1.0_
    1263ms  INFO c8_backup::elasticsearch Deleted index operate-import-position-8.2.0_
    1587ms  INFO c8_backup::elasticsearch Deleted index operate-user-1.2.0_
    1907ms  INFO c8_backup::elasticsearch Deleted index zeebe-record_deployment-distribution_8.2.3_2023-05-04
    2223ms  INFO c8_backup::elasticsearch Deleted index operate-operation-8.2.0_
    2509ms  INFO c8_backup::elasticsearch Deleted index zeebe-record_deployment_8.2.3_2023-05-04
    2809ms  INFO c8_backup::elasticsearch Deleted index operate-batch-operation-1.0.0_
    3077ms  INFO c8_backup::elasticsearch Deleted index operate-process-8.1.8_
    3352ms  INFO c8_backup::elasticsearch Deleted index operate-web-session-1.1.0_
    3634ms  INFO c8_backup::elasticsearch Deleted index operate-incident-8.2.0_
    3938ms  INFO c8_backup::elasticsearch Deleted index zeebe-record_variable_8.2.3_2023-05-04
    4216ms  INFO c8_backup::elasticsearch Deleted index zeebe-record_job_8.2.3_2023-05-04
    4472ms  INFO c8_backup::elasticsearch Deleted index operate-variable-8.2.0_
    4796ms  INFO c8_backup::elasticsearch Deleted index operate-migration-steps-repository-1.1.0_
    5069ms  INFO c8_backup::elasticsearch Deleted index operate-decision-1.0.0_
    5324ms  INFO c8_backup::elasticsearch Deleted index operate-event-8.1.0_
    5624ms  INFO c8_backup::elasticsearch Deleted index zeebe-record_process-instance-creation_8.2.3_2023-05-04
    5939ms  INFO c8_backup::elasticsearch Deleted index zeebe-record_process_8.2.3_2023-05-04
    6234ms  INFO c8_backup::elasticsearch Deleted index operate-metric-1.0.0_
    6526ms  INFO c8_backup::elasticsearch Deleted index operate-sequence-flow-8.2.0_
    6844ms  INFO c8_backup::elasticsearch Deleted index zeebe-record_process-instance_8.2.3_2023-05-04
    7164ms  INFO c8_backup::elasticsearch Deleted index operate-decision-requirements-1.0.0_
  
  c8_backup::restore::restore_indices{backup=Backup { id: 1683214620, snapshots: ["camunda_zeebe_records_1683214620", "camunda_operate_1683214620_8.2.3_part_1_of_6", "camunda_operate_1683214620_8.2.3_part_2_of_6", "camunda_operate_1683214620_8.2.3_part_3_of_6", "camunda_operate_1683214620_8.2.3_part_4_of_6", "camunda_operate_1683214620_8.2.3_part_5_of_6", "camunda_operate_1683214620_8.2.3_part_6_of_6"] }}
    12203ms  INFO c8_backup::restore Restored snapshot camunda_zeebe_records_1683214620
    14050ms  INFO c8_backup::restore Restored snapshot camunda_operate_1683214620_8.2.3_part_1_of_6
    18322ms  INFO c8_backup::restore Restored snapshot camunda_operate_1683214620_8.2.3_part_2_of_6
    18925ms  INFO c8_backup::restore Restored snapshot camunda_operate_1683214620_8.2.3_part_3_of_6
    23336ms  INFO c8_backup::restore Restored snapshot camunda_operate_1683214620_8.2.3_part_4_of_6
    23991ms  INFO c8_backup::restore Restored snapshot camunda_operate_1683214620_8.2.3_part_5_of_6
    26099ms  INFO c8_backup::restore Restored snapshot camunda_operate_1683214620_8.2.3_part_6_of_6
  
  c8_backup::restore::delete_zeebe_data{backup=Backup { id: 1683214620, snapshots: ["camunda_zeebe_records_1683214620", "camunda_operate_1683214620_8.2.3_part_1_of_6", "camunda_operate_1683214620_8.2.3_part_2_of_6", "camunda_operate_1683214620_8.2.3_part_3_of_6", "camunda_operate_1683214620_8.2.3_part_4_of_6", "camunda_operate_1683214620_8.2.3_part_5_of_6", "camunda_operate_1683214620_8.2.3_part_6_of_6"] }}
    303ms  INFO c8_backup::restore Deleting data of data-ccs23-dev-zeebe-0
    332ms  INFO c8_backup::restore Deleting data of data-ccs23-dev-zeebe-1
    360ms  INFO c8_backup::restore Deleting data of data-ccs23-dev-zeebe-2
    17284ms  INFO c8_backup::restore Deleted data of data-ccs23-dev-zeebe-0
    17385ms  INFO c8_backup::restore Deleted data of data-ccs23-dev-zeebe-1
    17482ms  INFO c8_backup::restore Deleted data of data-ccs23-dev-zeebe-2
  
  c8_backup::restore::restore_zeebe_data{backup=Backup { id: 1683214620, snapshots: ["camunda_zeebe_records_1683214620", "camunda_operate_1683214620_8.2.3_part_1_of_6", "camunda_operate_1683214620_8.2.3_part_2_of_6", "camunda_operate_1683214620_8.2.3_part_3_of_6", "camunda_operate_1683214620_8.2.3_part_4_of_6", "camunda_operate_1683214620_8.2.3_part_5_of_6", "camunda_operate_1683214620_8.2.3_part_6_of_6"] }}
    97ms  INFO c8_backup::restore Restoring data of data-ccs23-dev-zeebe-0
    131ms  INFO c8_backup::restore Restoring data of data-ccs23-dev-zeebe-1
    161ms  INFO c8_backup::restore Restoring data of data-ccs23-dev-zeebe-2
    26025ms  INFO c8_backup::restore Restored data of data-ccs23-dev-zeebe-0
    26117ms  INFO c8_backup::restore Restored data of data-ccs23-dev-zeebe-1
    26210ms  INFO c8_backup::restore Restored data of data-ccs23-dev-zeebe-2
  
  c8_backup::restore::start_apps{}
    0ms  INFO c8_backup::restore Starting apps
    29ms  INFO c8_backup::restore Started ccs23-dev-zeebe-gateway
    59ms  INFO c8_backup::restore Started ccs23-dev-operate
    90ms  INFO c8_backup::restore Started ccs23-dev-zeebe```