Sign in

Davinder Pal
Senior Software Engineer III ( R&D )

In this article, I will try to share the approach and Automation which can handle the upgrade of Apache Zookeeper.

I hope, you are already using automation created by me if not still the approach can help you do the job.

  1. Please don’t take this article as the only source of truth for the upgrade process, I am only sharing what I tested and done in the past which doesn’t imply that it will work 100% in your situation as well.
  2. Please read the official documentation/release notes for new versions as they might introduce breaking changes.
  3. Please test your approach…

Last Article in this series

Let’s continue, In this part, I will be covering few things about Ansible Module Utils aka common library or functions to reduce duplication of code.

It all started when I decided to write all the modules for AWS which were 190+ modules and almost all of these modules have consistent AWS API so you have to parse results and similarly show them.

Example Function

from ansible.module_utils.common.dict_transformations import camel_dict_to_snake_dictdef aws_response_list_parser(paginate: bool, iterator, resource_field: str) -> list:
_return = []
try:
if paginate…

Let’s continue from the last part :), please do a quick recap if you forgot something about the journey :D.

Let’s explorer behind the scene stuff for automatic documentation creation for Ansible Collection.

December 2020

Have you noticed that all the Ansible modules have a lot of components in the code itself like

Example Module: aws_amp.py

  1. DOCUMENTATION
  2. EXAMPLES
  3. RETURN

After Running the Documentation Tool, we get documentation like this
community.missing_collection.aws_amp_module.rst which far easier to read than actually reading the code.

Ansible Core Team had created a documentation tool that is/was generating documentation out of these values automatically whenever they were…


I suggest you should read the whole series or at least start with the previous one to get the full context. Below is the link for the previous article.

Nov 2020

What is Ansible Collection?
It is a set of similar modules which can do a couple of tasks for the given environment/region/cloud-provider/software/etc.

Why we need Ansible Collection?
Before Ansible 2.9.10 version, Ansible Modules were packed into Ansible Packages and All the modules have to meet a certain level of standard as they will be shipped with the official product and that lead serious issue which is the delay in the…


I hope you have read the first article for this series.

September 2018

First thing, we need to know about the module and its working, let’s take an example code and try to explain what it does for you?
Sample Code: mapr_service.py
you can treat the ansible module as a python script that does a very specific thing and it is exactly working as a script. Instead of running script manually with python3 <test-module>.py , we going to run the script with Ansible.

Ansible will handle a lot of things for us like

  1. Input / Outputs
  2. Formatting
  3. Compatibility across various…

This is a story of building 190+ Ansible Modules which are being later combined into one package called community.missing_collection.

Aug 2018

I was working on a Big Data Platform called “MapR” inspired by Apache Hadoop, It had a hell of a lot of components and those were earlier managed by MapR Cli Tool and It was hard to support it and do things with it. I was tasked to upgrade the platform and create/update Automation to do all this stuff because we had 10–15 clusters and I didn’t like to do it manually on all environments. I did manage to…


How I turned around things for myself and Make Success for Myself.

As I mentioned in my second article of this series, my PR was stuck and the Reviewer wasn’t ready to merge because of One line of Code.

Situation is still the same so far, Let’s not waste more time in that PR, it’s just too much for me.

Let’s talk about, how I learned something new out of it and made something useful for Ansible Community.

There was another bold thing happened in Ansible was to split modules into separate set of collections, some modules will be maintained…


I hope, you have read part 1 of it if not then I would definitely recommend you to read that as well to get the full context of it.

Let’s start with the next part of the story, this part will be quite interesting if you are an Ansible Community Developer.

last PR: https://github.com/ansible-collections/community.general/pull/876
New: https://github.com/ansible-collections/community.general/pull/1501

Finally, a reviewer was interested in reviewing the code after 2 years. He started reviewing my code in Sep 2020 and it continued till Dec 2020 but in Dec, Ansible Bot blocked my PR saying it should be rebased with ansible again so it…


It all started when I was creating a new AWS region deployment for a project, it took me a week to understand the process and implement it but that’s not important for this article, let’s move to an interesting topic. My Project was storing all its credentials to AWS Parameter Store only because it works with most of the tools and ease of implementation for storing sensitive data.

As per my memory, I never thought about DR of AWS Parameter store as was a very subtle topic and never came into my mind at least.

AWS Parameter store does have…


Honestly, there aren’t many good backup tools for Apache Kafka but why we need a backup tool in the first place. As Apache Kafka didn’t have any offsite backup solution or DR solutions for it, some tricks can be done but none are 100% reliable.

Example Trick:

create MM1 cluster and it will sync data one-way to another Apache Kafka Cluster Only.

My project/utility create offsite backups in local & cloud storage. Currently, it supports Any local Mounted Storage /AWS S3.

Project Source Code: 116davinder/apache-kafka-backup-and-restore

here is the sample for input configuration for AWS S3.

{
"BOOTSTRAP_SERVERS": "kafka01:9092,kafka02:9092,kafka03:9092"…

Davinder Pal

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store