Backup System for Apache Kafka ( Part 1 )

Honestly, there aren’t many good backup tools for Apache Kafka but why we need a backup tool in the first place. As Apache Kafka didn’t have any offsite backup solution or DR solutions for it, some tricks can be done but none are 100% reliable.

Example Trick:

create MM1 cluster and it will sync data one-way to another Apache Kafka Cluster Only.

Courtesy: freshmango.com

My project/utility create offsite backups in local & cloud storage. Currently, it supports Any local Mounted Storage /AWS S3.

Project Source Code: 116davinder/apache-kafka-backup-and-restore

here is the sample for input configuration for AWS S3.

How to Run Backup Application?

Expected Output:

Now leave this application running if you want to have continuous backups or close it. Now you can put extra policies on S3 as well to migrate data to S3 Glacier.

Notes*

  1. This application can be started in Container/Docker.
  2. This application can be started in SystemD.
  3. This application can be started multiple times to increase the speed of backup if required.
  4. This application supports UTF-8 based messages only.
  5. This application performs compression by default to tar.gz.
  6. This application creates sha256 hash files to ensure the integrity of backup files.
  7. Upload to S3 is based on RETRY_UPLOAD_SECONDS.

Hopefully, someone will not have to create a similar tool for their needs :).

Senior Software Engineer III ( R&D )