Dynamic inventory in Ansible with Hetzner Cloud

For the last few years, I've been using Ansible to configure as much of my self-hosted infrastructure as possible - whether it's servers in my physical possession, or a VPS from a cloud provider.

Ansible dynamic inventory

Previously, I had always manually defined all my hosts in an inventory file which might look something like this:

; inventory/hosts.ini

[webserver]
webserver1.example.net

Recently we rebuilt our Adventurous Way website, and in doing so decided to migrate it to new hosts in the Hetzner cloud. Rather than managing the inventory manually, I wanted to take advantage of Ansible's dynamic inventory capability.

Hetzner Cloud plugin

As part of the hetzner.hcloud collection, there is an inventory plugin called hetzner.hcloud.hcloud. This plugin will connect to Hetzner's API and retrieve a list of available hosts in a format that Ansible can use.

To start, I made sure to add the collection to my requirements.yml file and then ran ansible-galaxy install -r requirements.yml to install the collection.

# requirements.yml

collections:
  - name: hetzner.hcloud

Then, in my ansible.cfg I configured the location where I would define the inventory, in this case a folder called inventory in the root of my project. As an aside, I also specify where Ansible Galaxy should install roles and collections, then add the ./galaxy/ folder to my .gitignore file.

; ansible.cfg

[defaults]
inventory = inventory
roles_path = ./galaxy/roles:./roles
collections_path = ./galaxy/collections

Hcloud configuration

With the dependencies installed, configuring the plugin was as simple as adding the following to my inventory/hcloud.yml file - note the filename is important as that's what tells the plugin to run on that file.

# inventory/hcloud.yml

plugin: hetzner.hcloud.hcloud
token: 'YOUR_HETZNER_API_TOKEN_HERE'

As always, rather than hard-coding a secret API token, I wanted to use Ansible Vault. However, as I learned, you can't just add a secret to your vault here because the inventory plugin runs before your vault has been decrypted.

Instead, I used the inline vault encryption to give me an individually encrypted string I could add to my inventory configuration:

$ ansible-vault encrypt_string -n token "YOUR_HETZNER_API_TOKEN_HERE"

Encryption successful
token: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          37353934316337356533323233653333383863343230373337633964356439333361383034323733
          3130363039316436640a663536393061303239303537323463303935363339336165323134643665
          66303061383864306661613862303838346434323139336637333066356562363665

You can test this works by running ansible-inventory --graph to get a concise list of the hosts it has returned, or ansible-inventory --list to output all the host info.

Assigning roles

If your inventory is simple and you just need a flat list of all the hosts, then you've got it.

However, a powerful feature of Ansible inventory is being able to group the hosts together. Using the dynamic inventory plugin, you can add labels to the hosts in the Hetzner cloud, and then use those to dynamically assign roles.

A trivial example of this is to assign a label called role to a server, then add the following configuration to your hcloud.yml file:

# inventory/hcloud.yml

keyed_groups:
  - key: labels.role
    prefix: role

Resulting in something like:

$ ansible-inventory --graph
@all:
  |--@ungrouped:
  |--@hcloud:
  |  |--database1.example.net
  |  |--webserver1.example.net
  |  |--webserver2.example.net
  |--@role_database:
  |  |--database1.example.net
  |--@role_webserver:
  |  |--webserver1.example.net
  |  |--webserver2.example.net

Connecting via Tailscale

Once I have a host configured, I like to use the Hetzner firewall to block all ports except those I explicitly allow through - typically I only allow TCP on ports 80 (HTTP) and 443 (HTTPS).

Ansible requires access to a machine via SSH on TCP port 22.

Rather than opening up port 22 to the world, I use Tailscale to create a Wireguard based mesh VPN. I can then connect my own machine to Tailscale, and access my server over the secure VPN.

The only problem is that the dynamic inventory plugin returns the external IPv4 address of each Hetzner cloud host, and it has no knowledge of the fact Tailscale is even running on the host, let alone the IP address.

Given I'm managing a relatively small number of hosts, I chose to solve this a very simple way. During my bootstrapping process for a new host, I add a label to the host in Hetzner called ssh_ip with its value set to whatever the Tailscale IP address is. I do this manually but it could easily be automated (and I'd like to do so in future).

I can then use the compose feature in the dynamic inventory configuration to read this label and insert its value into the ansible_host variable (which is what Ansible tries to connect to):

# inventory/hcloud.yml

compose:
  ansible_host: "'{{ labels.ssh_ip | default(ipv4) }}'"

One thing to watch for - note the double and single quotes above. It's important you include this, otherwise it won't work.

You could also use this exact same mechanism to set any other variables based on labels, essentially migrating a lot of host-based configuration from your host_vars into the Hetzner cloud. If used appropriately, this can be a powerful feature.

Wrapping up

All my Hetzner inventory is now dynamically configured when I run Ansible, polling the Hetzner API for the information. By adding the Tailscale IP address to each host as a label, I also ensure that I have secure access to each host via the Wireguard VPN without having to expose any unnecessary ports to the internet.

Even if you only manage a handful of hosts, you may find this to be a powerful and convenient way to avoid your Ansible inventory configuration becoming too difficult to manage.

While I used Hetzner for my needs, this same technique should work with any of the myriad of providers supported by Ansible's dynamic inventory plugins.