Journey of Apache Kafka & Zookeeper Administrator ( Part 16 )

In this article, I will try to share the approach and Automation which can handle the upgrade of Apache Zookeeper.

https://codeflex.co

I hope, you are already using automation created by me if not still the approach can help you do the job.

116davinder/zookeeper-cluster-ansible

1. Please don’t take this article as the only source of truth for the upgrade process, I am only sharing what I tested and done in the past which doesn’t imply that it will work 100% in your situation as well.
2. Please read the official documentation/release notes for new versions as they might introduce breaking changes.
3. Please test your approach in your test environment, don’t touch your production environment because it seems easy to do.
4. This article doesn’t cover anything related to Zookeeper on Containers / Kubernetes. Please refer to your original vendor documentation or similar documentation.

Let’s try to do it like a Real Administrator will do aka Manually.

The approach is to update one server/instance at a time and so on aka Rolling Update.

1. Download Latest Version or required version tar from the Apache Zookeeper website.
2. Unpack the tar into the server directory where you want to store the packages of the zookeeper.
3. Let’s generate the configuration for the new version.
   Here, please do read the release notes of Zookeeper because they might have removed old configs or added some new configurations.
   we have to generate the following configs files as per my automation.
   1. zoo.cfg
   2. java.env
   3. myid
   4. log4j.properties
4. Please do make sure that myid value doesn’t change because it will create problems for the apache zookeeper.
5. Update the symlink to the new version of apache zookeeper folder as this will help us easy rollback.
6. Restart systemd zookeeper service and it will start using the new version of apache zookeeper.
7. Check Logs as per your log dir settings in the log4j settings or whatever is passed as a command-line option.
8. Check Logs of Other Zookeeper nodes as well, if they can recognize new zookeeper instance or not.
9. Check Monitoring scripts are working fine or not.
10. Now, let’s repeat this whole process for all of the apache zookeeper nodes.

If you several environments in your company then you have to do this process for each of your apache zookeeper servers.

The above-mentioned process works perfectly fine if your company has less than 3 clusters with every 3 nodes but it will be more and more painful once the number of clusters grows and the upgrade of zookeeper increases that’s where my automation comes into the picture and saves your precious time from doing redundant tasks so you can focus on something much better or learn something new that can help you grow instead of spending hours in just upgrading zookeeper clusters.

My automation has an ansible playbook that does almost all of the above tasks for you and does it much faster.

116davinder/zookeeper-cluster-ansible

Automation won’t work if you have created your clusters manually because it follows a very specific structure to manage zookeeper packages and other things which may not work if you have manually created clusters.

The last thing, please don’t take these notes and automation for granted, I am human so I do make mistakes as well so it's your job to test your clusters before and after the upgrade.