Journey of the development of Ansible Collection? (Part 5)

Journey of the development of Ansible Collection? (Part 4)

https://auth0.com

Let’s continue, In this part, I will be covering few things about Ansible Module Utils aka common library or functions to reduce duplication of code.

It all started when I decided to write all the modules for AWS which were 190+ modules and almost all of these modules have consistent AWS API so you have to parse results and similarly show them.

Example Function

from ansible.module_utils.common.dict_transformations import camel_dict_to_snake_dict
def aws_response_list_parser(paginate: bool, iterator, resource_field: str) -> list:
    _return = []
    try:
        if paginate:
            for response in iterator:
                for _app in response[resource_field]:
                    try:
                        _return.append(camel_dict_to_snake_dict(_app))
                    except AttributeError:
                        _return.append(_app)
        else:
            for _app in iterator[resource_field]:
                try:
                    _return.append(camel_dict_to_snake_dict(_app))
                except AttributeError:
                    _return.append(_app)
        return _return
    except KeyError:
        return _return

Before this function, I was writing this if-for-for thing in all the modules and it was getting hard to test them in every module and which affected my speed of development to its knees so I decided to create a function which has shown above which helped me a lot but still I had to copy-paste this function in all modules which duplicated and it was very hard to update the function, as I found more use-cases and better handling of errors so I had to update this function all my modules which created quite a lot of work for me.

I already knew that ansible modules support util functions that can be imported from certain paths but never put effort into discovering those hidden secrets.

When I reached 100+ modules and each of them had the same function and found a couple of bugs in this function then I realized that It was too painful for me to update this function all 100+ modules every time, when I find a bug for this common function in future so I decided to put a pause on development and started researching util functions. Luckily research paid off and found a way you can import these common functions in modules within a collection.

Let me bring back the collection structure.

$ tree -L 2
.
├── docs
├── easy_dev.sh
├── galaxy.yml
├── LICENSE
├── meta
│   └── runtime.yml
├── plugins
│   ├── modules
│   └── module_utils
├── README.md
├── requirements.txt
└── tests
    ├── integration
    └── readme.md

As you can see there is a folder called plugins/modulee_utils that is the correct place to put any common function/library for your collection. Now here is the important thing to understand.

As module_utils and modules are sitting next to each other doesn’t mean that we can import them just with relative paths.

None working Code for Me

from xxx import yyy
..module_utils.xx_file import yy_function

Here is a better version of the code

from xxx import yyy
from ansible_collections.community.missing_collection.plugins.module_utils.aws_response_parser import aws_response_list_parser

Now, Ansible knows from where it needs to import the code for the import statement and that’s the whole story in one line :).

Finally, I removed the duplicate helper function from all modules and it removed 4.5k lines of code in total and Now it is far easier for me to update the common function if there is any bug or feature that needs to be implemented.

Thanks a lot for reading the article with one line solution at last :)

I will be writing the last piece of this puzzle, which is Tests!