I use Ansible to automate just about everything I can in my homelab. I have roles ranging from installing a custom root certificate to configuring a complete Bareos backup system, and everything in between, and the rapid expansion of my playbooks has presented some challenges.

One big issue is that I’ve always just developed roles directly against a target system, iterating on the role until the target system is exactly how I want it and then committing what I have. The next time I run the role, half the time it doesn’t work. Ansible’s idempotence means that as I grow a role, what I’ve already done isn’t being tested again in a now more complex environment, so when it’s run against a clean system, it doesn’t behave the same. This leads to small errors and a lack of idempotence but also means I can easily miss really obvious things, like forgetting to install the necessary dependencies on a clean system. Lastly, roles grow stale over time, and eventually, I end up installing outdated versions of software or failing to install dependencies because the old downloads have been removed.

I’m hoping that Molecule can solve some of these problems. Molecule creates customizable “execution environments” to test your roles in, either using a container engine or full-fat VMs on a local hypervisor or in the cloud. It’s pretty easy to get started with, and makes developing roles a lot easier and faster.

Getting Started with Molecule

Molecule can be installed using pip, by running pip install molecule molecule-plugins. The molecule-plugins package isn’t strictly necessary, by default you can use the delegated driver and write your own playbook to deploy infrastructure, but the default plugins package add support for Docker, Podman, and others.

Once Molecule is installed, it’s easy to create a new Molecule-enabled role or add a scenario to an existing role. I’m mostly adding Molecule to existing roles for now, so I used the command molecule init scenario -r time -d containers to add a Molecule scenario for my time role, which handles setting system timezones across multiple platforms. The command created a new molecule folder within my role, shown below.

time
├── molecule
│   └── default
│       ├── converge.yml
│       ├── molecule.yml
│       ├── prepare.yml
│       └── verify.yml
└── tasks
    └── main.yml

Actually, I added the prepare.yml file later, but the other three files were created by Molecule. The molecule.yml file configures the Molecule stages, which include the prepare, converge, and verify stages, which are all simply Ansible playbooks. The molecule.yml contains the following sections:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

---
dependency:
  name: galaxy
driver:
  name: containers
platforms:
  - name: time
    image: docker.io/geerlingguy/docker-ubuntu2204-ansible:latest
    pre_build_image: true
provisioner:
  name: ansible
  inventory:
    group_vars:
      all:
        timezone: America/Los_Angeles
verifier:
  name: ansible

Note the driver, which is containers. I use Podman to test my roles in a rootless container, but using the containers driver instead of podman allows Molecule to use either Docker or Podman, depending on which one is available (I believe it has a preference for Docker if both are installed). I’ve also had to edit the default molecule.yml slightly to be able to test this specific role.

First, most of my environment uses Ubuntu Server, so roles are primarily tested on a docker-ubuntu2204-ansible container, which adds Ansible’s dependencies onto a base Ubuntu container image. Second, I added a group_vars section to set a variable which would usually be set in the file group_vars/all.yml. Molecule doesn’t inherit host or group variables, since it’s an isolated execution environment which is only aware of itself and variables from imported roles.

Adding Preparation Steps

For most of my roles, the base docker-ubuntu2204-ansible container just isn’t enough. Although it packs more than a base Ubuntu container, it’s missing most of the packages needed by my roles. Heck, even the very simple time role doesn’t work because the tzdata package isn’t installed. To fix this, I created a prepare.yml Molecule playbook. This playbook is executed immediately after the container is created, when running the molecule create command. I could also create a custom Dockerfile, which might be a bit faster since the prepare.yml playbook has to be run every time a container is spun up, but in my experience Molecule tried to rebuild the container every time anyway so it didn’t save any time. If you wanted to use a custom container image, Molecule supports a dockerfile option:

 6
 7
 8
 9
10

platforms:
  - name: custom
    dockerfile: Dockerfile.j2
    image: docker.io/geerlingguy/docker-ubuntu2204-ansible:latest
    pre_build_image: false

Dockerfile.j2 can be a Jinja2 template, which can read from an item variable which represents the specific instance from the platforms array:

1
2
3

FROM {{ item.image }}

RUN apt update && apt install -y tzdata

However, I preferred the cleanliness of using one container image rather than customizing it for every role. Using a prepare.yml playbook also lets you use Ansible modules instead of shell commands:

1
2
3
4
5
6
7
8
9

---
- name: Prepare
  hosts: all
  tasks:
    - name: Install dependencies
      ansible.builtin.apt:
        name: tzdata
        state: present
        update_cache: true

Converging and Verifying Roles

Most of Molecule’s power comes from the converge.yml and verify.yml files. The converge.yml file is a normal playbook, which by default imports the role to be tested, although you can add any other Ansible tasks here as well:

1
2
3
4
5
6
7

---
- name: Converge
  hosts: all
  tasks:
    - name: Include time
      ansible.builtin.include_role:
        name: time

Once the converge.yml playbook completes, the role can be tested, by performing assertions on the Molecule instance. Again, the verify.yml file is a simple Ansible playbook, which uses the assert module to test the state of the converged system:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

---
- name: Verify
  hosts: all
  gather_facts: false
  tasks:
    - name: Read timezone
      ansible.builtin.slurp:
        src: /etc/timezone
      register: timezone

    - name: Assert that timezone was updated
      ansible.builtin.assert:
        that: timezone.content | b64decode | trim == "America/Los_Angeles"

And that’s all that’s needed to set up Molecule testing for a role!

Running Molecule

With Molecule fully configured, the role can be tested just by running the molecule test command. The command creates and prepares the container, then runs the Ansible role against it during the converge stage, and lastly runs the verify playbook to see if the system reached the desired state. Lastly, it destroys the container and exits. This is particularly useful in a CI environment, since it performs every step with just one command, but Molecule is equally powerful for developing and iterating on roles.

By running molecule create you can spin up and prepare the container, and leave it running to continuously test against. You can run molecule converge and molecule idempotence (did I mention Molecule can run your playbook multiple times to test for idempotence?) to test the role as many times as needed, and use molecule login to spawn a shell in the container to troubleshoot issues with. Lastly, you can write assertions in verify.yml and run these with molecule verify, and then run molecule destroy to tear down the infrastructure (or to start fresh with a new container).

Testing Complex Roles

I’ve only just gotten started developing and testing with Molecule, but using containers, while fast, imposes some limitations on what you can test with the barebones configuration. For example, by default containers do not run an init system such as systemd, so testing roles which install a systemd unit won’t work, and that tends to be most of them.

Luckily, most of these restrictions can be worked around. For example, to start the container with systemd, the container needs to be privileged and have cgroup access, which can be accomplished using a volume mount. Then, systemd can be started as PID 1 by calling the /usr/sbin/init script:

 6
 7
 8
 9
10
11
12
13

platforms:
  - name: time
    image: docker.io/geerlingguy/docker-ubuntu2204-ansible:latest
    pre_build_image: true
    privileged: true
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    command: /usr/sbin/init

I assume this could be done similarly with device nodes like /dev/net/tun, which is a bridge I’ll have to cross eventually for another role. That being said, I haven’t found myself spending too much time on Molecule instead of writing the roles themselves. Out of the box, it works how you need it to, and it uses common tools like Docker and Podman for which there is extensive documentation when you run into a limitation you didn’t expect.