Enter the automation dream with Ansible and AWX
Dive into the world of automation and discover how it can clear your mind and make your IT work much more organized and peaceful.
Automaception
I try to approach my work—managing IT infrastructures—as relaxed as possible. Paradoxically, that means I spend a lot of time optimizing my work. Fortunately, machines are ideal for that (the game Factorio takes up more time than I care to admit). Computers even more so, which is probably half the reason I work in this field. So when I was introduced to Ansible, I was immediately sold.
It enabled me to let computers manage themselves at the touch of a button. You can neatly organize infrastructure configuration into groups and roles, and the community provides collections to manage a huge number of components. Optimizing the Ansible book and ensuring that the server park is correctly configured and that changes are implemented by the servers in an organized manner is a wonderful challenge. Completing (part of) such an Ansible book feels like pure bliss: you run the playbook and the controller does all the work for you. And you? You have plenty of time to sit back and watch the automation.
But that's exactly where a slight itch slowly starts to develop. It begins in the back of your mind, in a sarcastic tone:
Don't you just love pressing buttons to make computers do things?
And at first you don't understand it, but then it adds:
Wouldn't it be nice if you could automate that too?
And so a vicious circle is created.
So, naturally, a genius (or rather, several geniuses) thought of automating the automation, and thus AWX was born (the upstream version of the Ansible Automation Platform). But that's not enough, of course: we also automate the automation that automates the automation—the configuration of the controller. This is the story of a possible path you can take, past all those turtles.
Disclaimer: You don't need to know much about AWX and/or the infra.controller_configuration collection to read this article. But if you plan to follow my path, I recommend that you do familiarize yourself with it. In any case, it helps if you have a conceptual understanding of how Ansible works.
I'll skip the beginning of my journey and drop you a few months later, at the point where the itch in my head became too loud to ignore. At that time, the team I'm part of managed about 500 virtual and physical servers with one large Ansible book. We were manually running so many playbooks that it was driving us crazy. We had just started running AWX and were in the middle of the migration when a keyboard-loving colleague complained loudly:
I feel like I've reverted to clicking in the AWX web interface instead of actually automating anything!
Everyone knew: a comment like that meant he was onto something. And indeed—it turned out to be two things:
- Managing the actual Ansible code and projects stored and executed in AWX
- Managing AWX itself and the executions of that code
In this article, I will focus on managing AWX itself. I will leave the migration to another time—and perhaps to another writer.
Manage AWX
This new layer of automation that we have rolled out has its own configuration stored in a database. You can modify this configuration via the AWX API or by clicking in the web interface. However, these are manual actions—and we prefer to avoid them. The infra.controller_configuration collection helps with this by providing an Ansible way to manage the AWX configuration.
We are going to configure AWX and all associated components using an Ansible playbook. This resulted in a playbook that configures the following parts of AWX, among others:
-
Global AWX settings, such as login settings, LDAP configuration (including global admins and auditors), and general properties of this AWX instance.
-
Organizations, teams, and their roles within organizations.
-
Execution Environments, because the standard environment really only works for a proof of concept.
-
Container groups, because the default setting rarely works well in practice.
-
Credentials, all of which originate from HashiCorp Vault.
-
Projects, loaded from Git repositories, of course.
-
Inventories, including variables, also from Git repositories.
-
Job templates, so you can roll out changes in phases instead of all at once.
-
Schedules, to actually run those jobs.
-
Notifications, so that when jobs fail, a Jira issue is automatically created and we receive a notification in our chat application.
If we were still on the first "turtle," we would write all this code, integrate it into our large Ansible book, and execute it together with the rest. But we had already gone one level deeper, so we placed all this code in a separate Ansible book. We ran that, after which AWX executed the large Ansible book. It was bliss.
Unfortunately, that bliss was short-lived. Not long after, AWX 'broke'. In other words: someone changed the AWX configuration described above, deployed a new 'feature', made a mistake, and everything came to a standstill.
The team quickly came up with a neat process to prevent this: someone else had to review and approve the changes before they were integrated into our Ansible books. For major changes, two colleagues would even review them. A wonderful process—with one small problem: we were back to square one. We were once again spending enormous amounts of time double and triple checking code. This deeper turtle level suddenly seemed a lot less useful.
But if Inception has taught us one thing, it's that you can always go one level deeper. So we descended to the next turtle.
That's where GitOps comes in—more specifically, GitLab (read the closing remarks if you're surprised we weren't already using it). We created a GitLab project, uploaded the Ansible book with the AWX code, and built a CI/CD pipeline to free us from manual work. That pipeline consists of the following steps:
-
Checks our YAML files for correct syntax with yamllint.
-
Checks the Ansible code for pitfalls with ansible-lint.
-
Scan for potential vulnerabilities with GitLab's automated security scans.
-
Performs a dry run of the entire code.
-
Deploy the change to our staging AWX instance.
-
Deploy the change to production, but only after explicit approval and planning.
The keen reader will notice that there is still a manual action involved. That's correct. But we no longer have to worry about syntax errors, typos, or simple bugs—and all without pressing any buttons. Thanks to a staging AWX instance, we can also easily test whether a change breaks anything, instead of just thinking about it theoretically.
In addition to no longer having to perform this ourselves or wait for deployments, our change management has also improved significantly:
- Each change is linked to a Jira issue.
- A majority of the team must approve, and that approval is recorded.
- There is more time to inform end users about upcoming changes.
- Reversing is easy.
- Security and compliance scans provide additional insights to improve our infrastructure.
- Code quality reports and test results are automatically available to the compliance department.
Conclusion
This new way of working with Ansible not only offers the team a more robust way to manage the infrastructure, but also provides greater scalability. In addition to the benefits for ourselves, the organization also gains better insight into the status of the underlying infrastructure.
But the most important result is, of course, this: I can finally sit back and watch our servers manage themselves. It's a wonderful feeling—a job well done. My team and I now have time to think about new features or help the organization improve itself further. Does something need to be adjusted? Then I modify the code, commit it to GitLab, and watch the magic unfold. A true technological marvel.
…
“Don't you love pressing buttons to make computers do things?”
…
“Wouldn't it be nice if you could automate that too?”
Will I ever be able to silence that annoying little voice in my head? Or could AI finally quiet that voice for me?
Final Thoughts
This blog post is based on an actual implementation I carried out for a client, but some parts have been modified to better fit the story. A few comments on these modifications:
- GitOps was already in use, but has been further expanded and well integrated rather than implemented entirely from scratch.
- Many peripheral issues have been simplified or omitted; otherwise, this article would quickly have turned into a white paper.
- In my opinion, implementing secret management in AWX is not a trivial matter and requires a great deal of attention and care.
- The way changes in your Ansible books affect your projects and jobs is an interesting but challenging puzzle in itself.
AWX is only supported via the AWX Operator on Kubernetes. Those are dragons—you have been warned. But to be fair, they are incredibly cool.
