Set up a Galera Witness on Hetzner using Terraform + Ansible

Download Markdown

Table of Contents

1 set up a galera witness on

In this tutorial, we will automate deployment of a VPS on Hetzner (a renowned EU cloud-based provider) using Terraform and configure it using Ansible - all triggered from AWX. This is a definitive guide for anyone running Galera on hybrid cloud infrastructure!

1. What are we trying to achieve here?

AWX will run the following three templates as part of one workflow:

With a click of a button, you will be able to spin up a fully configured VPS for < 4 EUR a month in an environment of a trusted European-based cloud provider and be able to re-create it whenever or wherever needed 🙂

The result that we aim for - a 3-template workflow to provision your VPS in Hetnzer (or another provider)

Use case - what is this setup for?

  • You have Site 1 and Site 2 already configured using a MariaDB Galera Cluster and web servers.
  • Site 1 and Site 2 are connected using a Site-to-site VPN tunnel (WireGuard), each with equal weight in terms of Galera weighting.
  • Site 3 will therefore act as a ‘witness’ in between the two sites - if one goes down, the DB cluster will remain operational.

Proposed architecture:

  • 1st Site/Segment 1 (Uvody) - main site:
    • Proxmox host 1 - 192.168.8.3/24
      • Galera-A1 LXC - Weight - 1 - 192.168.8.71/24
      • Galera-A2 LXC - Weight - 1 - 192.168.8.72/24
      • Web1 VM - 192.168.8.9/24
      • OPNSense 1 VM (CARP - master, HAProxy, S2S VPN) - 192.168.8.1/24 (LAN CARP = 192.168.8.254/24), S2S VPN on 10.10.10.1/24
      • AWX-Ansible VM (K3s) - 192.168.8.25/24
      • Uptime Kuma LXC - 192.168.8.60/24
    • Proxmox host 2 - 192.168.8.4/24
      • Gitea LXC - 192.168.8.20/24
      • Galera-A3 LXC - Weight - 1 (Proxmox host 2) - 192.168.8.73/24
      • Galera-A4 LXC - Weight - 1 (Proxmox host 2) - 192.168.8.74/24
      • Galera-template LXC (offline) - ID 150 - backed up to PBS
      • Web2 VM - 192.168.8.10/24
      • OPNSense 2 VM (CARP - backup, HAProxy, S2S VPN) - 192.168.8.2/24
    • Proxmox Backups Server (PBS, runs on RPI 4) - 192.168.8.17/24
  • 2nd Site/Segment 2 (Tusarka) - fallback site:
    • Proxmox host 3 - 192.168.6.4/24
      • Galera-A5 - Weight 2 (Proxmox host 3) - 192.168.6.75/24
      • Galera-A6 - Weight 2 (Proxmox host 3) - 192.168.6.76/24
      • Web3 VM - 192.168.6.11/24
      • OPNSense 3 (HAProxy, S2S VPN) - 192.168.6.1/24, S2S VPN on 10.10.10.2/24
  • 3rd Site/Segment 3 (Hetzner VPS) - services running on the VPS:
    • WireGuard S2S VPN - S2S VPN on 10.10.10.3/24
    • Galera witness - Weight - 1 (S2S VPN)
    • Uptime Kuma - monitor websites, VPN tunnel + Galera via push scripts

For an easier visualization, here is a flowchart diagram:

3 proposed architecture

Existing setup this tutorial assumes (Site 1 + Site 2)

  • Site 1 and 2 are already assumed to be set up from before and are not covered by this tutorial.
  • Galera weight by Segment: 4+4+1=9 - required min. quorum: 5 (1 full site + witness)
    • Outage happens only if both Site 1 AND Site 2 are down. Alternatively, if the witness node is down and Site 1 OR Site 2 goes down.
  • Web VMs sync web data via Gitea (a separate LXC) and user data using syncthing
  • CloudFlare load balancing prioritizes Site 1 and fallbacks to Site 2 during an outage
  • OPNSense on each site provides sticky binding to a web VM host with an inserted cookie
  • UptimeKuma LXC is located in each site to monitor local services (web1, web2, galera cluster health, VPN tunnel, etc.).
    • It is also installed on the witness node to monitor the S2S VPN tunnel + overall web services (e.g. ‘does the website load?’).
  • The bucket serves as a backup location for catastrophic situations since Site 1 and Site 2 are physically only 1 mile away from each other.

Galera set up on Site 1 & 2 - weights & segments

While the Site 1 & 2 configuration is out of the scope of this tutorial, we have already explored which firewall ports will need to be opened and how to troubleshoot them (at least on OPNSense).

  • In case you have been running your Galera cluster on just one site for now, you may have noticed that the config file for the Arbitrator contains a term ‘segment’. What is it?

    • Galera segments allow you to group nodes based on their physical location (e.g., Site 1, Site 2, Cloud). By assigning these segments, you enable Optimized WAN Replication: instead of the primary node sending individual data packets to every remote node over your VPN tunnel (which consumes massive bandwidth), it sends a single copy to one node in the remote segment, which then acts as a relay to distribute the data locally to its neighbors. This drastically reduces traffic across your site-to-site links and prevents database replication from choking your network.

  • In its default setting, each node carries one weight. In the proposed architecture section earlier, you may have noticed that Site 2 has only 2x Galera nodes, whereas Site 1 has 4x nodes. So how can you influence weighting on the galera nodes?

  • An example for Site 2’s Galera node config where we need to set:

    • segment: 2
    • weight for each node: 2
sudo nano /etc/mysql/mariadb.conf.d/60-galera.cnf

[galera]
# ... your other settings ...

# Site 2 Specifics: Segment 2, Weight 2
wsrep_provider_options="gcache.size=512M;gcs.fc_limit=128;gcs.fc_factor=0.8;**gmcast.segment=2;pc.weight=2**"

# Full Cluster List
wsrep_cluster_address="gcomm://192.168.**8**.71,192.168.**8**.72,192.168.**8**.73,192.168.**8**.74,192.168.**6**.75,192.168.**6**.76"

# Node Specifics (Example for A5)
wsrep_node_address = "192.168.6.75"
wsrep_node_name = "galera-a5"
wsrep_sst_receive_address = "192.168.6.75"
  • Similarly, on Site 1, ensure you use:
    • weight: 1 (since we have 4 nodes)
    • segment: 1 (as our first site)
[galera]
# ... your other config ...
wsrep_provider_options="gcache.size=512M;gcs.fc_limit=128;gcs.fc_factor=0.8;gmcast.**segment=1;pc.weight=1**"
  • Once set up, SSH into any Galera node in your cluster and compare the values:
mysql -u root -p

# With 4+4+1 design, you should see 9
SHOW STATUS LIKE 'wsrep_cluster_weight';

# With 4 nodes + 2 nodes + 1 witness, you should see 7
SHOW STATUS LIKE 'wsrep_cluster_size';

Other Infrastructure Pre-requisites:

  • These you can have running on either of your Sites as VMs or containers.
  • A Gitea server (or some other source version control service hosted locally or externally).
  • AWX (or plain Ansible if you are comfortable running playbooks manually). What will you need for the execution environment (keep in mind that this means both the modules in requirements.yml as well as related dependencies in requirements.txt - check out this part of my previous guide).
    • hetzner.hcloud - to provision the VPS and firewall rules on Hetzner
    • community.general - for CloudFlare DNS record changes, Terraform
    • ansible.netcommon - sudo elevation on the VPS
    • awx.awx - to add the VPS into our AWX inventory
    • google.cloud - to interface with a GCP bucket
    • community.docker - to work with a Docker container for UptimeKuma
  • An account in Hetzner - we will be deploying a VM for 5 EUR a month
  • Reverse proxy and load balancer on Site 1 + 2 (I use OPNSense in this tutorial)
  • WireGuard Site-to-Site VPN set up on Site 1 + 2 already
  • CloudFlare account - to automate managing DNS records for Site 3 as the public IP of the instance may change during re-provisioning.
  • For the third template, I will have a kuma.db (backup) of Uptime Kuma to restore from a GCP bucket.

2. Create the initial Terraform + Ansible scripts

Firstly, we will create our scripts and based on that, we will create the credential types and add the credentials in. This will hopefully help shed some light on what we are doing and why.

  • Scripts required to provision the VPS:
    • main.tf: This file is the main set of instructions. It defines the provider, finds the right OS image, and creates the server with your exact specifications. See Terraform manual.
    • 1a-provision-witness-terraform.yml - use Terraform to provision the VPS in Hetzner (before cloud-init below is used).
    • 1b-provision-witness-hetzner.yml - the cloud-init script that will install and configure services on the VPS. See Hetzner’s manual.
    • outputs.tf: This file tells Terraform what information to print out when it's done. This is critical for AWX integration.

First script - main.tf:

  • Provider Setup: Configures the hcloud (Hetzner Cloud) provider to manage resources.
  • Image Selection: Automatically finds the latest available image for Debian 13 (Trixie) on x86 architecture.
  • Server Creation: Provisions a CX23 VPS (2 vCPU, 4GB RAM) in the Falkenstein data center (fsn1).
  • Cloud-Init Injection: Injects the 1b-provision-witness-hetzner.yml user data to handle OS-level setup immediately upon boot.
  • Firewall Management: Creates and attaches a robust Hetzner Cloud Firewall (witness_fw) that strictly limits ingress traffic to SSH, WireGuard, and Uptime Kuma ports (this is external to ufw that also gets installed on the VPS later using the 1b template).
**#** **main.tf
**
# This tells Terraform we are using the Hetzner Cloud provider
terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1" # Use the latest 1.x version
    }
  }
}

# The provider will automatically use the HCLOUD_TOKEN environment variable
provider "hcloud" {}

# ----------------------------
# --- DEFINE VM PROPERTIES ---
# ----------------------------

# This data block finds the latest "debian-13" image ID
data "hcloud_image" "debian_image" {
  name        = "debian-13"
  with_architecture = "x86"
}

# This data block finds your SSH key to add to the server.
data "hcloud_ssh_key" "jan_key" {
  name = "Jan's key 2025-06"
}
data "hcloud_ssh_key" "ansible_key" {
  name = "Ansible"
}

# This is the main resource block that creates the VM
resource "hcloud_server" "witness_vm" {
  name        = "galera-witness"
  server_type = "cx23"                # 2 vCPU, 4GB RAM, 40GB SSD
  image       = data.hcloud_image.debian_image.id
  location    = "fsn1"                # Falkenstein (eu-central)

  # Enable/disable ipv4 and ipv6
  public_net {
    ipv4_enabled = true
    ipv6_enabled = false
  }

  # Add your SSH key for initial access (before cloud-init runs)
  ssh_keys = [
    data.hcloud_ssh_key.jan_key.id,
    data.hcloud_ssh_key.ansible_key.id
  ]

  # This reads the cloud-init for Hetzner and passes it to the server
  user_data = file("1b-provision-witness-hetzner.yml")

  labels = {
    "service" = "galera"
    "role"    = "witness"
  }
}

# -------------------------------------------------
# --- DEFINE THE HETZNER FIREWALL AND ITS RULES ---
# -------------------------------------------------
resource "hcloud_firewall" "witness_fw" {
  name = "galera-witness-fw"

  # Rule 1: Allow SSH (on your new port) from anywhere
  rule {
    direction = "in"
    protocol  = "tcp"
    port      = "2222"
    source_ips = [
      "0.0.0.0/0",
      "::/0"
    ]
  }

  # Allow WireGuard (UDP) from Site 1 and Site 2
  rule {
    direction = "in"
    protocol  = "udp"
    port      = "51821"
    source_ips = [
      "0.0.0.0/0",
      "::/0"
    ]
  }

  # Rule 3: Allow Galera (TCP/UDP) from  VPN subnets
  rule {
    direction = "in"
    protocol  = "tcp"
    port      = "4567"
    source_ips = [
      "192.168.0.0/16",
      "10.10.10.0/24"
    ]
  }
  rule {
    direction = "in"
    protocol  = "udp"
    port      = "4567"
    source_ips = [
      "192.168.0.0/16",
      "10.10.10.0/24"
    ]
  }

  # Rule 4: Allow ICMP (Ping)
  rule {
    direction = "in"
    protocol  = "icmp"
    source_ips = [
      "0.0.0.0/0",
      "::/0"
    ]
  }

  # Rule 5: Allow Uptime Kuma (TCP) from anywhere
  # Later, this can be restricted to the Site 1 + 2 and other WG Roadwarrior IP addresses
  rule {
    direction = "in"
    protocol  = "tcp"
    port      = "3001"
    source_ips = [
      "0.0.0.0/0",
      "::/0"
    ]
  }
}

# -----------------------------------------
# --- ATTACH THE FIREWALL TO THE SERVER ---
# -----------------------------------------
resource "hcloud_firewall_attachment" "fw_attachment" {
  firewall_id = hcloud_firewall.witness_fw.id
  server_ids  = [hcloud_server.witness_vm.id]
}

Terraform provisioning script:

  • Terraform Execution: Runs terraform apply to provision the actual infrastructure on Hetzner.
  • Dynamic Inventory: Captures the newly created server's public IP address from Terraform's output.
  • AWX Integration: Automatically adds the new host to the AWX Hetzner inventory, allowing subsequent job templates to target it immediately without manual intervention.
  • DNS Automation: Updates a CloudFlare DNS A-record (hetzner-witness.bachelor-tech.com) to point to the new IP, ensuring VPN endpoints remain valid even if the IP changes.
# 1a-provision-witness-terraform.yml

---
- name: Provision Hetzner Witness VM with Terraform
  hosts: localhost
  connection: local
  gather_facts: no

  tasks:
    - name: Run Terraform to create the witness server
      community.general.terraform:
        project_path: "{{ playbook_dir }}"
        state: present              # This means "run terraform apply"
        force_init: true            # This runs "terraform init" first

      # This is how the playbook gets the Hetzner token
      # from the AWX credential (see step 4)
      environment:
        HCLOUD_TOKEN: "{{ lookup('env', 'HCLOUD_TOKEN') }}"

      # This registers the output of the 'terraform apply' command
      register: tf_output

    - name: Show the Witness IPv4 Address
      ansible.builtin.debug:
        msg: "Server '{{ tf_output.outputs.witness_id.value }}' created with IPv4: {{ tf_output.outputs.witness_ipv4.value }}"

    - name: Add new VM to AWX Inventory
      awx.awx.host:
        name: "galera-witness-hetzner"
        inventory: "Hetzner"  # Or whatever your inventory is called
        variables:
          ansible_host: "{{ tf_output.outputs.witness_ipv4.value }}"
          ansible_port: 2222
          ansible_user: ansible
        state: present

      environment:
        # Token for AWX API - adjust your hostname, as required
        CONTROLLER_HOST: "{{ lookup('env', 'TOWER_HOST') | default('https://awx.bachelor-tech.com', true) }}"
        CONTROLLER_OAUTH_TOKEN: "{{ lookup('env', 'AWX_TOKEN') }}"
        CONTROLLER_VERIFY_SSL: false # Set to true if you have valid SSL

    - name: Update CloudFlare DNS record
      community.general.cloudflare_dns:
         zone: "bachelor-tech.com"
         record: "hetzner-witness"
         type: "A"  # A record is for IPv4
         value: "{{ tf_output.outputs.witness_ipv4.value }}"
         api_token: "{{ cloudflare_api_token }}"

      no_log: true # Hides the token from the log output

Cloud-config file:

  • Adds two users (ansible user for management via S2S VPN with AWX from Site 1)
  • Custom SSH port (with no password auth) + installs packages apart from the one used for Galera
  • Installs Docker + configures ufw firewall rules
  • The cloud-config file (the file MUST start with the #cloud-config line or else it will not be recognized and the following will not be applied):
**# 1b-provision-witness-hetzner.yml**

#cloud-config

# Add users
users:
  - name: jan
    groups: users, admin
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS+K109p5/R9YjsGrzW5smURig7pOF+ex3BoBW5a9ZISUQW7A9vdOavwHbGZC5oCM7DyexZwQhr1BVdlLFW4X8= ecdsa-key-20250630
  - name: ansible
    groups: users, admin
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICYSPfWeFXXskRcuSCa8valFv0rCZY8RizM+68MYMa9c

# Install the pre-requisites for adding the repo
package_update: true
packages:
  - curl
  - gpg
  # Required for Docker:
  - ca-certificates
  - gnupg
  - python3-pip
  - mariadb-client # To check for Galera cluster size
package_upgrade: true

# Write into the SSH config file
write_files:
  - path: /etc/ssh/sshd_config.d/ssh-hardening.conf
    content: |
      PermitRootLogin no
      PasswordAuthentication no
      Port 2222
      KbdInteractiveAuthentication no
      ChallengeResponseAuthentication no
      MaxAuthTries 2
      AllowTcpForwarding no
      X11Forwarding no
      AllowAgentForwarding no
      AuthorizedKeysFile .ssh/authorized_keys
      AllowUsers jan ansible      

# Run setup commands
runcmd:
  # Apply the new SSH port
  - systemctl restart sshd

  # Manually add the MariaDB repo (from which we will fetch the arbitrator package)
  - curl -o /etc/apt/keyrings/mariadb-keyring.pgp https://mariadb.org/mariadb_release_signing_key.pgp
  # Fetch MariaDB 11.8.5 compatible with Trixie
  - echo "deb [signed-by=/etc/apt/keyrings/mariadb-keyring.pgp] https://deb.mariadb.org/11.8.5/debian trixie main" > /etc/apt/sources.list.d/mariadb.list
  # Update and install the packages
  - apt-get update
  - apt-get install -y fail2ban ufw mc wireguard wireguard-tools rsync galera-arbitrator-4

  # Configure them
  - printf "[sshd]\nenabled = true\nport = ssh, 2222\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
  - systemctl enable fail2ban
  - systemctl start fail2ban

  # --- Install Docker ---
  - install -m 0755 -d /etc/apt/keyrings
  - curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.gpg
  - chmod a+r /etc/apt/keyrings/docker.gpg
  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian trixie stable" > /etc/apt/sources.list.d/docker.list
  - apt-get update
  - apt-get install -y python3-docker docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  # Add ansible user to docker group
  - usermod -aG docker ansible
  - systemctl enable docker
  - systemctl start docker

  # Configure UFW
  - ufw allow 2222/tcp # SSH
  - ufw allow 51821/udp # Site-to-site VPN listening port
  - ufw allow 3001/tcp # UptimeKuma's web interface
  - ufw allow from 192.168.0.0/16 to any port 4567 # Allows the remote LAN to reach the Arbitrator
  - ufw allow from 10.10.10.0/24 to any port 4567 # Site-to-site VPN for Galera Arbitrator

  # Enable UFW
  - ufw --force enable

The Outputs file:

  • Exposes Data: Defines exactly which data points (IP addresses, Server ID) Terraform should return to Ansible after the provisioning completes.
  • Integration Key: This allows the Ansible playbook to read tf_output.outputs.witness_ipv4.value and use it to update DNS and Inventory. The ipv6 address is provided as an optional extra for those who would prefer to use that, instead (in which case, modify the 1a script as well).
# outputs.tf

output "witness_ipv4" {
  description = "The public IPv4 address of the witness server."
  value       = hcloud_server.witness_vm.ipv4_address
}

output "witness_ipv6" {
  description = "The public IPv6 address of the witness server."
  value       = hcloud_server.witness_vm.ipv6_address
}

output "witness_id" {
  description = "The ID of the witness server."
  value       = hcloud_server.witness_vm.id
}

3. Prepare your AWX environment

In order to be able to execute these templates saved in Gitea, we will need to prepare the environment in AWX.

Create an API token to provision Hetzner-based resources using Terraform

  • Create an account with Hetzner and activate 2FA.
  • Create a token:
    • Go to the Hetzner Cloud Console.
    • Select your project (and note the ID from the browser URL, you will need it later).
    • On the left, click Security.
    • Select the API Tokens tab.
    • Click Generate API Token.
    • Give it a name (e.g., terraform-provisioner) and set its permissions to Read & Write.
    • Click Generate API Token.
    • Copy the token. You will not see it again.

Automate adding your Hetzner VPS into AWX inventory after creation

  • Once the VPS is set up in Hetzner, the outputs.tf will help us to get the public ipv4 or ipv6 address of the host and we will be able to add the host into our inventory to manage it later.

  • Create an AWX API Token:

    • In AWX UI, go to your User Profile (bottom left).
    • Click on the Tokens tab.
    • Click Add and create a new Personal Access Token.
    • Give it a Scope of Write.
    • Copy the token immediately! You will not see it again.

  • Now, let's store this token securely. Create the credential

    • Go to Administration -> Credential Types.
    • Click Add and create a new Credential Type.
    • Name: AWX Controller Token
    • Input & Injector Configuration - copy paste it as below:
# Input configuration
fields:
  - id: AWX_TOKEN
    label: AWX Token
    type: string
    secret: true    

# Injector Configuration:
env:
  AWX_TOKEN: '{{ AWX_TOKEN }}'

4 automate adding your hetzner

  • With the credential type created, go to Resources -> Credentials. Add a new credential:
    • Name: AWX API Token
    • Credential Type: AWX Controller Token
    • AWX Token: Paste your new token here.
    • Save it.

Create your Hetzner token + virtual local host to run jobs from

  • Create the Hetzner Credential. Go to Administration -> Credential Types and create the Hetzner Cloud Token type:
# Input Configuration
fields:
  - id: HCLOUD_TOKEN
    label: Hetzner API Token
    type: string
    secret: true

# Injector Configuration
env:
  HCLOUD_TOKEN: '{{ HCLOUD_TOKEN }}'

5 create your hetzner token

  • Then go to Resources -> Credentials and create a new credential using this type. Paste in your Hetzner API token.

  • In AWX UI, go to Resources → Inventory and select ‘Add inventory’ from the dropdown.

    • Name: Hetzner
    • Save it and move to the Hosts tab. Add a new host.
    • Name: localhost
    • Variables: copy paste the row below:
---
ansible_connection: local

Credentials for updating DNS records in CloudFlare for Hetzner-created VPS

In order to set up the Site to Site VPN with Site 1 and Site 2 later on, providing the fact that the IP address issued by Hetzner may change during (re-)provisioning of the VPS, we should maintain a DNS record that Site 1 and Site 2 use to connect to Site 3’s endpoint and update it dynamically upon the VPS creation. Unless you have a CloudFlare token already set up, follow these steps to create it.

Create a CloudFlare token

  • In your CloudFlare dashboard, go to My Profile -> API Tokens.
  • Click Create Token.
  • Use the "Edit zone DNS" template.
  • Set the permissions as follows:
    • Permissions: Zone - DNS - Edit
    • Zone Resources: Include - Specific zone - bachelor-tech.com
  • Create the token and copy it, as you will not see it again.

Create a New AWX Credential Type

  • Go to Administration -> Credential Types and click Add.
  • Name: CloudFlare API Token
  • Add in the following input & injector configuration:
# Input configuration:
fields:
  - id: cloudflare_api_token
    label: CloudFlare API Token
    type: string
    secret: true

# Injector configuration:
extra_vars:
  cloudflare_api_token: '{{ cloudflare_api_token }}'

Create the New Credential for CloudFlare

  • Go to Resources -> Credentials and click Add.
  • Name: CloudFlare (bachelor-tech.com)
  • Credential Type: Select your new CloudFlare API Token.
  • CloudFlare API Token: Paste your new token.

4. Create a provisioning job in AWX [Template 1]

In case you have created the YAML and Terraform files in a new repo, you will need to ensure that they are pulled into AWX. If you are working on an existing repo, then just run a sync job.

Add Your Gitea Project (if not done already):

  • Go to Resources -> Projects.
  • Click Add.
  • Name: Gitea - Infrastructure
  • Source Control Type: Git
  • Source Control URL: Your Gitea repo URL.
  • Add your Gitea credential (you can create it in Gitea as read only - see my previous guide)
  • Click Save and let it sync.

Create the Job Template (after a sync from Gitea):

  • Go to Resources -> Job Templates.
  • Click Add -> Add Job Template.
  • Name: H1. Provision Galera Witness - Hetzner
  • Job Type: Run
  • Click on the Credentials tab. Add the AWX API Token credential and the Hetzner API Token credential and the CloudFlare API token (so 3 in total!).
  • Project: Select your Gitea - Infrastructure project.
  • Playbook: provision-witness.yml (This should pop up from your Gitea project).
  • Execution Environment: Select your existing environment (see here for a step-by-step guide on how to create one - you will need modules like community.general for Terraform to work).
  • Inventory: Select the localhost item we created earlier.
  • Click Save and Launch it.

6 create the job template after

  • It’s such a nice feeling to see the automation works for you!

7 create the job template after

  • The output from the template log running - what we see in there:
    • The IP address was provided.
    • The VPS was added into inventory.

8 create the job template after

  • Now, check the inventory in AWX!

9 create the job template after

Troubleshooting the 1st template

Lots of things can go wrong, starting from little YAML-related syntax mistakes to certain commands not working on your distro or version of choice.

  • Potential issues before the VPS is created:

    • Missing community.general in your EE → cannot call Terraform
    • Missing hetzner.hcloud in your EE → cannot interface with Hetzner
    • Missing ansible_connection: local in your localhost host results in errors related to the inability to match a host
    • Your cloud-config file (provision-witness-hetzner.yml) does not start with the #cloud-config line on the first line, making it non-recognizable when executed on the VPS.
    • Cloud init script issues → one part completes and the rest does not - check the logs above in detail.
    • Firewall rules in Hetzner do not get created → check the syntax against the official TF Hetzner documentation in case there have been changes.
    • Re-running the template fails - have you manually removed both the VPS and the firewall rules in Hetzner?
    • Or the other way - if you get Error: name is already used (uniqueness_error, 97cc8f7bf626fbe0084738ed7d6b0cdd) with hcloud_firewall.witness_fw, it means that you are running the job again, removed the VPS but forgot to remove the firewall rule in Hetzner.
    • During re-creation of the VPS, you forgot to remove the firewall rules, making the job fail.
    • Unable to find your host or script - ensure that you have synced your project playbook (synced from Gitea).
    • MariaDB repo issues - check the newest version for your distro on MariaDB’s mirror.
    • Firewall-related errors - such as when you enter a hostname and only IP addresses are accepted.

  • Some challenges you may run after the VPS creation:

    • Whatever error you may experience once the VPS is created, you can find it in one of the two log files called cloud-init.log and cloud-init-output.log. Let me know in the comments below if you get stuck or would like to understand what the errors in the logs mean (unless you prefer to use AI to interpret for you!).
    • To get to those logs, open your respective project in Hetzner and go to the details about your server.
    • Go to the Rescue tab and scroll down to click on the ‘Reset Root Password’ button.
    • A new root password is generated, save it.
    • Use the console button in the top right area to console it. Enter root for the username and manually type in the generated password. You will get into the VPS. Check the following logs:
# The output from cloud-init (the initial commands)
sudo nano /var/log/cloud-init.log

# The output of the RUN commands
sudo nano /var/log/cloud-init-output.log

5. Create a Configuration job template in AWX [Template 2]

We now have our VPS in our inventory and can reach its public IP address via SSH to run additional jobs. What we want is to automate the set up of a Site 2 Site VPN with the other two sites + to configure garbd, so that our VPS can start acting as a witness.

Prepare WireGuard configuration template

  • Create the second template to configure the VPS in Hetzner:
    • Pre-flight Checks: Waits for the SSH port (2222) to become available and ensures cloud-init has finished installing all base packages.
    • VPN Configuration: Generates the WireGuard configuration file (wg0.conf) using Jinja2 templates and secrets from AWX, then enables the service.
    • Galera Arbitrator: Deploys the garb configuration, sets up log rotation for the arbitrator logs, and starts the service to join the cluster.
**# 2-configure-witness.yml**

---

- name: 1. Verify Witness is Ready
  hosts: galera-witness-hetzner
  gather_facts: no # Don't try to gather facts until we know it's online

  pre_tasks:
    - name: Wait for SSH port (2222) to be available
      ansible.builtin.wait_for:
        host: "{{ ansible_host | default(inventory_hostname) }}"
        port: "{{ ansible_port | default(2222) }}"
        state: started
        delay: 5      # Wait 5s before first check
        timeout: 300  # Wait up to 5 minutes
      delegate_to: localhost # Run this check from the AWX container
      become: false # No need for sudo

    - name: Wait for cloud-init to finish
      ansible.builtin.command:
        cmd: cloud-init status --wait
      changed_when: false
      become: true # This must run with sudo

- name: 2. Configure WireGuard on Witness
  hosts: galera-witness-hetzner
  become: true

  tasks:
    - name: Ensure /etc/wireguard directory exists
      ansible.builtin.file:
        path: /etc/wireguard
        state: directory
        owner: root
        group: root
        mode: '0700' # drwx------

    - name: Create WireGuard wg0.conf
      ansible.builtin.template:
        src: wg0.conf.j2
        dest: /etc/wireguard/wg0.conf
        owner: root
        group: root
        mode: '0600'
      notify: Restart wireguard

    - name: Ensure WireGuard starts on boot
      ansible.builtin.systemd_service:
        name: wg-quick@wg0
        enabled: yes
        state: started

  handlers:
    - name: Restart wireguard
      ansible.builtin.systemd_service:
        name: wg-quick@wg0
        state: restarted

- name: 3. Configure Galera Arbitrator (garbd)
  hosts: galera-witness-hetzner
  become: true

  tasks:
    - name: Create and set permissions for garbd.log
      ansible.builtin.file:
        path: /var/log/garbd.log
        state: touch
        owner: nobody
        group: nogroup
        mode: '0644'

    - name: Create garb configuration
      ansible.builtin.template:
        src: garb.default.j2
        dest: /etc/default/garb
        owner: root
        group: root
        mode: '0644'
      notify: Restart garb

    - name: Add logrotate configuration for garb
      ansible.builtin.copy:
        dest: /etc/logrotate.d/garb
        content: |
          /var/log/garbd.log
          {
              daily
              rotate 7
              compress
              delaycompress
              missingok
              notifempty
              create 0644 nobody nogroup
          }          
        owner: root
        group: root
        mode: '0644'

    - name: Ensure garbd starts on boot
      ansible.builtin.systemd_service:
        name: garb  # The service name is called garb, not garbd
        enabled: yes
        state: started

  handlers:
    - name: Restart garb
      ansible.builtin.systemd_service:
        name: garb
        state: restarted

WireGuard configuration file:

  • Interface Definition: Configures the local WireGuard interface with the private key injected securely from AWX credentials.
  • Peer Setup: Defines the connection details (Endpoint, Public Key, AllowedIPs) for Site 1 and Site 2, establishing the mesh VPN topology.
**# wg0.conf.j2**

[Interface]
# This is the witness node's configuration
Address = {{ witness_wg_ip | default('10.10.10.3/24') }}
ListenPort = 51821
PrivateKey = {{ witness_wg_private_key }}

# --- Peer 1: Site 1 - U vody (OPNSense) ---
[Peer]
PublicKey = {{ site1_wg_public_key }}
Endpoint = {{ site1_wg_endpoint | default('uvody.bachelor-tech.com:51821') }}
AllowedIPs = 192.168.8.0/24, 10.10.10.1/32

# --- Peer 2: Site 2 - Tusarka (OPNSense) ---
[Peer]
PublicKey = {{ site2_wg_public_key }}
Endpoint = {{ site2_wg_endpoint | default('tusarka.bachelor-tech.com:51821') }}
AllowedIPs = 192.168.6.0/24, 10.10.10.2/32

Garb configuration for the Galera arbitrator:

  • Cluster Config: Defines the Galera cluster address string (GALERA_NODES), listing all other nodes in the cluster so the arbitrator knows who to connect to.
  • Arbitrator Mode: Sets specific options (like gmcast.segment) to ensure the witness participates in voting and is treated as a separate segment for latency reasons.
**# garb.default.j2**

# Configuration for Galera Arbitrator
# This file is sourced by /usr/bin/garb-systemd

# Cluster name from your 60-galera.cnf
GALERA_GROUP="clusterA"

# List of ALL *DATA NODES* (Sites 1 & 2)
GALERA_NODES="192.168.8.71:4567,192.168.8.72:4567,192.168.8.73:4567,192.168.8.74:4567,192.168.6.75:4567,192.168.6.76:4567"

# Set the segment for this witness node
GALERA_OPTIONS="gmcast.segment=3"

# Log file location
LOG_FILE="/var/log/garbd.log"

Generate a Site 3 public/private keys

  • This only needs to be done once when you create it for the first time.
  • On any Linux machine that has WireGuard installed, run the following. This will provide two files, witness_private.key and witness_public.key. We will store the private key in its own credential type.
wg genkey | tee witness_private.key | wg pubkey > witness_public.key
  • As for the public key, you can add it into your Site 1 and Site 2’s WG configuration. In my case, I have OPNSense running with WireGuard, so I would add it as a peer in there (for each Site 1 and Site 2’s OPNSense):
    • Enabled: tick
    • Name: Hetzner_Witness_peer
    • Public key: paste in your key
    • Pre-shared key: leave blank
    • Allowed IPs: 10.10.10.3/32 (just the interface)
    • Endpoint address: either the public IP or a hostname that is kept up to date with a dynamic DNS client.
    • Endpoint port: 51821
    • Instances: your local WG S2S instance

10 generate a site 3

  • Create a Custom Credential Type
    • In AWX, go to Administration -> Credential Types. Add a new one.
    • Name: WireGuard Private Key
    • Add the Input and Injector Configuration:
# Input Configuration
fields:
  - id: private_key
    label: WireGuard Private Key
    type: string
    secret: true

# Injector Configuration
extra_vars:
  witness_wg_private_key: '{{ private_key }}'

11 generate a site 3

  • Now go to Resources -> Credentials. Add a new one
  • Name: Witness WG Private Key
  • Credential Type: Select your new WireGuard Private Key type.
  • WireGuard Private Key: Paste in the contents of your witness_private.key file.

Firewall rules on Site 1 + Site 2

In order for the Galera Arbitrator communication to occur from Site 3 with Site 1+2, we need to open a port on the VPN tunnel to pass traffic on TCP and UDP port 4567. Here is an example with OPNSense that is located on Site 1 and Site 2. You will need to apply this rule on each.

  • On OPNSense, go to Firewall → Rules → WireGuard S2S interface (whatever you have called it). Add a new rule:
    • Action: Pass
    • Interface: WG S2S VPN
    • Direction: in
    • TCP/IP: IPv4
    • Protocol: TCP/UDP
    • Source: 10.10.10.3/32 (Site 3 VPN)
    • Destination: LAN net
    • Port: other - 4567 to 4567 (this is the port that garb uses, unlike SQL)
    • Log: tick Log packets that are handled by this rule
    • Description: Allow Galera Witness In

12 firewall rules on site 1 site

  • Save and apply the rule.
  • Then if you have ufw (or another local firewall service like iptables) running on each of your galera nodes, you will need to open ports for the communication with the Arbitrator over the S2S VPN to work on port 4567 TCP+UDP:
    • If using ufw, SSH into each Galera node (Site 1 + Site 2) and run the following using the IP of the tunnel of Site 3:
sudo ufw allow from 10.10.10.0/24 to any port 4567 proto tcp
sudo ufw allow from 10.10.10.0/24 to any port 4567 proto udp

Create a new job template in AWX for Configuring the VPS

  • Name: H2 - Configure Galera Witness
  • Inventory: Hetzner
  • Project: Gitea’s or your preferred source version control system
  • Limit: Add your witness VPS, such as galera-witness-hetzner
  • Execution environment: same as for your first template
  • Playbook: 2-configure-witness.yml - if you do not see it, sync your playbook from the Project section first to fetch it from Gitea.
  • Credentials: ansible - the SSH key we use to log into VMs and Site 3 WG private key
  • Add these extra variables:
---
# These are all public and safe to store as plain text
site1_wg_public_key: "PASTE_SITE1_PUBLIC_KEY_HERE"
site1_wg_endpoint: "site1:51821"
site2_wg_public_key: "PASTE_SITE2_PUBLIC_KEY_HERE"
site2_wg_endpoint: "site2:51821"
witness_wg_ip: "10.10.10.3/24"
  • Tick the box for Privilege Escalation

13 create a new job template in

  • Give it a test and run it!

Troubleshooting the 2nd template

  • Remember that if you choose to run the 1st template from start, you will need to remove both the VPS and the firewall rules in Hetzner before re-running it.
  • If you experience errors during the first part of the playbook run, ensure that the variable names in Gitea match the ones in the template variables.
  • In case you get stuck, post the relevant parts of the output log in the comments below and we can troubleshoot it together.
  • While the 2nd template is probably the simplest from the three, the most likely hiccup you may experience is with the site-to-site VPN communication and firewall ports. Let’s confirm that it works.

Verify that Galera comms is established

  • Once communication is established with the Site 3 Arbitrator, you can run this command from any Galera node (not the Arbitrator) to reveal the cluster size - the number should be increased by one:
mysql -u root -p

SHOW STATUS LIKE 'wsrep_cluster_size';
  • Observe the traffic under Firewall → Log Files → Live view on port 4567, you can observe the traffic coming in and out once the garb (or garbd) service is up on Site 3.

14 verify that galera comms is

6. Create an Uptime Kuma restoration job [Template 3]

In this workflow, we are automating the final piece of a Galera Cluster deployment. We have already provisioned a VPS (Hetzner) and configured VPN networking with garb Arbitrator service. Now, we need to deploy Uptime Kuma to this node to act as a local monitor.

💡 Note

Crucially, we are not starting from scratch. We are restoring a backup from Google Cloud Storage (GCS) so our monitoring history and settings are preserved, and we are configuring the node to monitor itself immediately upon boot. In other words, you will need a pre-prepared kuma.db SQLite file that can be fetched.

Pre-requisites for the Galera Monitor service in Uptime Kuma

  • In order to observe the status of the cluster (e.g. how many nodes are active), we will need to have set up a system user with limited privileges in order to reach one DB in the cluster (pick the most stable one).
  • You will therefore need to set it up only on one node, but it is recommended to apply it on all or a template that you use to ensure you can reach them in the future.
  • SSH into any of the Galera nodes (since they sync users and their credentials between each other) and run these commands - use the IP range that you use for your S2S VPN, as that is the incoming IP that the cluster will see:
mysql -u root -p

-- Create a user that can connect from the VPN subnet (10.10.10.x)
CREATE USER 'kuma_monitor'@'10.10.10.%' IDENTIFIED BY 'YOUR_SECURE_PASSWORD';

-- Grant minimal access (USAGE is enough to check status)
GRANT USAGE ON *.* TO 'kuma_monitor'@'10.10.10.%';
FLUSH PRIVILEGES;
  • Also, if you run ufw on the nodes (or a similar local firewall service like iptables), ensure that the VPS can reach the node(s) for monitoring of the cluster size. It is best to monitor one node from each site or all of them, if you prefer.
# Allow SQL connections from the WireGuard VPN subnet
sudo ufw allow from 10.10.10.0/24 to any port 3306 proto tcp
  • Add the credentials for the kuma_monitor into AWX. Go to Administration -> Credential Types and click Add.
  • Name: Galera Monitor User
  • Add the input configuration & injector configuration below:
# Input configuration:
---
fields:
  - id: username
    type: string
    label: Database Username
  - id: password
    type: string
    label: Database Password
    secret: true

# Injector configuration:
---
extra_vars:
  db_monitor_user: '{{ username }}'
  db_monitor_pass: '{{ password }}'

  • Then to save content of the credentials, go to Resources -> Credentials and click on the Add button.

    • Name: Witness DB Monitor
    • Credential Type: Select Galera Monitor User (the one you just created).
    • Username: kuma_monitor
    • Password: (Enter your secure password)

  • Lastly, ensure that on both your Site 1 and Site 2 firewall, the port 3306 is opened on the site tunnel VPN interface. For example, in my case, on OPNSense, go to Firewall → Rules → choose the WireGuard S2S VPN interface and click on the + sign to add a new rule:

    • Action: Pass
    • Interface: WG_S2S VPN
    • Direction: in
    • Interface: IPv4
    • Protocol: TCP
    • Source: 10.10.10.3/32 (the IP of the tunnel on the VPS side)
    • Destination: LAN net
    • Destination port: 3306 to 3306
    • Log packets: tick
    • Description: Allow SQL kuma_monitor in

15 pre requisites for the galera

Create a service account in GCP & add it as a custom credential type

  • To reach your bucket with a backup of an Uptimekuma database to restore, you would need a service account from GCP. These steps show how to do it, although you may already have one set up from before:
  • Navigate to GCP: Go to the GCP Console -> APIs and service -> Credentials.
  • Click + Create credentials and select Service account from the drop-down menu
    • Name: awx-gcs-reader (or similar).
    • Service account ID: It will auto-populate.
    • Click on the Create and Continue button.

16 create a service account in

  • Grant Roles:

    • Refresh the Credentials page to display your new service account. Open it and click on the Permissions tab. Then click on the Manage access button.
    • Add role: Storage Object Viewer and Storage Bucket Viewer (beta), which is needed for metadata.
    • This role provides read-only access to GCS objects, which is all we need.
    • Click on the Save button.

  • Generate a JSON Key:

    • Find your new awx-gcs-reader account in the list of Credentials and click on it.
    • Go to the Keys tab.
    • Click on Add key -> Create new key.
    • Select JSON as the key type and click Create.
    • Your browser will download a JSON file. Keep this file secure; you'll need its contents soon.

  • Next, we teach AWX how to understand this new GCP JSON key. Navigate to AWX: Go to Administration -> Credential Types and click on the Add button.

    • Name: GCP Service Account
    • Copy paste the content from the Input Configuration & Injector configuration fields.
# Input configuration:
fields:
  - id: service_account_json
    type: string
    label: Service Account JSON
    secret: true

# Injector configuration    
env:
  GCP_SERVICE_ACCOUNT_CONTENTS: '{{ service_account_json }}'
  • Now we can save the credential. Navigate to AWX: Go to Resources -> Credentials and click on the Add button.
    • Name: GCP UptimeKuma Backup
    • Credential Type: Select your new GCP Service Account from the list.
    • Service Account JSON: Open the JSON file you downloaded from GCP before. Copy the entire contents and paste them into this field. Click on the Save button.

Uptimekuma restoration template + Push scripts to monitor Galera cluster

Here is how this Ansible playbook (3-restore-uptimekuma.yml) works, step-by-step.

# 3-restore-uptimekuma.yml

---

- name: 1. Download Backup from GCS (on AWX)
  hosts: localhost
  connection: local
  gather_facts: no
  become: false

  vars:
    gcs_bucket: proxmox-backup-bachelor 
    gcs_object_path: "hetzner-backup/kuma.db"
    local_temp_backup: "/tmp/kuma.db"
    gcs_project_id: "113447253568"

  tasks:
    - name: Download Uptime Kuma backup from GCS
      google.cloud.gcp_storage_object:
        action: download
        bucket: "{{ gcs_bucket }}"
        src: "{{ gcs_object_path }}"      # object name in the bucket
        dest: "{{ local_temp_backup }}"   # the local file path
        project: "{{ gcs_project_id }}"
        auth_kind: serviceaccount
        service_account_contents: "{{ lookup('env', 'GCP_SERVICE_ACCOUNT_CONTENTS') }}"
      register: gcs_download

    - name: Verify download
      ansible.builtin.debug:
        msg: "Successfully downloaded {{ gcs_object_path }} to {{ local_temp_backup }}"
      when: gcs_download.changed

- name: 2. Restore and Run Uptime Kuma (on Galera Witness VPS)
  hosts: galera-witness-hetzner
  become: true # Run tasks below as root

  vars:
    kuma_data_dir: /opt/uptimekuma
    db_monitor_hosts: "192.168.8.71 192.168.6.73"
    local_temp_backup: "/tmp/kuma.db" # Path on the AWX controller
    garb_push_url: "{{ 'http://127.0.0.1:3001/api/push/YOUR_GARB_PUSH_TOKEN'
                    + '?status=up&msg=Garb%20service%20is%20up&ping=' }}"
    cluster_push_url: "{{ 'http://127.0.0.1:3001/api/push/YOUR_CLUSTER_PUSH_TOKEN'
                       + '?status=up&msg=ok&ping=' }}" # The status message will vary and we will modify it later

  tasks:
    - name: Install System Tools (rsyslog, mariadb-client)
      ansible.builtin.apt:
        pkg:
          - rsyslog
        state: present
        update_cache: yes

    - name: Ensure rsyslog is running
      ansible.builtin.service:
        name: rsyslog
        state: started
        enabled: yes

    - name: Create Uptime Kuma data directory
      ansible.builtin.file:
        path: "{{ kuma_data_dir }}"
        state: directory
        owner: root
        group: root
        mode: '0755'

    - name: Copy backup file from AWX to Witness
      ansible.builtin.copy:
        src: "{{ local_temp_backup }}"
        dest: "{{ kuma_data_dir }}/kuma.db" # This restores the backup
        owner: root
        group: root
        mode: '0644'

    - name: Get the host system timezone # Required for fail2ban to work in Docker for uptimekuma
      ansible.builtin.command: cat /etc/timezone
      register: host_tz_output
      changed_when: false # Don't report this as a "change" in the summary

    - name: Start Uptime Kuma container
      community.docker.docker_container:
        name: uptimekuma
        image: louislam/uptime-kuma:2 # Version 2.x.x
        state: started
        pull: true # Always pull it in case there is an updated version
        restart: true  # Restart the service after the DB is restored
        restart_policy: always
        ports:
          - "3001:3001"
        volumes:
          # This maps the host dir (with your .db) into the container
          - "{{ kuma_data_dir }}:/app/data"
          - "/etc/timezone:/etc/timezone:ro" # Sync time with host
          - "/etc/localtime:/etc/localtime:ro"          
        tty: false             # Tells app 'you are not in a terminal'
        interactive: false     # Disables interactive mode
        env:
          TZ: "{{ host_tz_output.stdout }}"
          NO_COLOR: "1" # to avoid binary output in systemd to ensure logs will flow to fail2ban
          FORCE_COLOR: "0"     # stronger disable for Node.js/Chalk
        log_driver: syslog # Avoid using systemd as it cannot process blob data from Node
        log_options:
          tag: uptimekuma  # Tag the logs so Fail2Ban can find them

    - name: Create the garb check script
      ansible.builtin.copy:
        dest: /usr/local/bin/check_garb.sh
        mode: '0755'
        owner: root
        group: root
        content: |
          #!/bin/bash
          # Managed by Ansible - Do Not Edit Manually

          PUSH_URL="{{ garb_push_url }}"
          SERVICE="garb"

          # Check if service is active
          if systemctl is-active --quiet "$SERVICE"; then
              # Service is UP. Send heartbeat.
              # -m 10: Max 10 seconds wait
              curl -fsS -m 10 "$PUSH_URL" > /dev/null 2>&1
          fi          

    - name: Add cron job for garb check
      ansible.builtin.cron:
        name: "Check Galera Arbitrator"
        minute: "*/2" # Run every two minutes
        job: "/bin/bash /usr/local/bin/check_garb.sh"
        user: root
        state: present

    - name: Create the Cluster Size check script (High Availability) for UptimeKuma
      ansible.builtin.copy:
        dest: /usr/local/bin/check_cluster_size.sh
        mode: '0755'
        owner: root
        group: root
        content: |
          #!/bin/bash
          # Managed by Ansible
          # This script checks multiple nodes for redundancy

          HOSTS="{{ db_monitor_hosts }}" # This injects the db_monitor_hosts variable
          DB_USER="{{ db_monitor_user }}"
          DB_PASS="{{ db_monitor_pass }}"
          PUSH_URL_BASE="{{ cluster_push_url | split('?') | first }}"

          SIZE=""

          # Loop through the hosts
          for HOST in $HOSTS; do
              # Try to get the size. 
              # -s: Silent, -N: Skip headers
              # --connect-timeout=3: Fail fast (3 seconds) if node is down
              TEMP_SIZE=$(mariadb -h $HOST -u "$DB_USER" -p"$DB_PASS" -s -N --connect-timeout=3 -e "SHOW STATUS LIKE 'wsrep_cluster_size';" 2>/dev/null | awk '{print $2}')

              if [[ -n "$TEMP_SIZE" ]]; then
                  SIZE=$TEMP_SIZE
                  # We got a valid number, stop looking!
                  break
              fi
          done

          # Check if we got a result from ANY node
          if [[ -z "$SIZE" ]]; then
              # If we are here, ALL nodes failed to respond
              echo "Cluster unreachable"
              curl -fsS "$PUSH_URL_BASE?status=down&msg=Connection%20Failed%20(All%20Nodes)&ping=" > /dev/null
              exit 1
          fi

          # Logic: Is size >= 5?
          if [ "$SIZE" -ge 5 ]; then
              # STATUS OK
              curl -fsS "$PUSH_URL_BASE?status=up&msg=Cluster%20Size:%20$SIZE&ping=" > /dev/null
          else
              # STATUS DEGRADED
              curl -fsS "$PUSH_URL_BASE?status=down&msg=Degraded%20Size:%20$SIZE&ping=" > /dev/null
          fi          

    - name: Add cron job for Cluster Size check
      ansible.builtin.cron:
        name: "Check Galera Cluster Size"
        minute: "*/2" # Run every two minutes
        job: "/bin/bash /usr/local/bin/check_cluster_size.sh"
        user: root
        state: present

    # Create the Filter for Fail2ban to recognize Uptime Kuma logs
    - name: Create Fail2Ban filter for Uptime Kuma
      ansible.builtin.copy:
        dest: /etc/fail2ban/filter.d/uptimekuma.conf
        owner: root
        group: root
        mode: '0644'
        content: |
          [Definition]
          # Since Node.JS 'colors' the output, it is safer to use these phrases
          # You can simulate failed logins and then check: sudo tail -f /var/log/syslog | grep uptimekuma
          # Match lines regardless of color codes or prefixes
          # We search for "[AUTH]" and the specific error messages
          failregex = Incorrect username or password.*IP=<HOST>
            Invalid token provided.*IP=<HOST>
            Too many failed requests.*IP=<HOST>
          ignoreregex =          
    # Create the jail for Uptime kuma
    - name: Create Fail2Ban jail for Uptime Kuma
      ansible.builtin.copy:
        dest: /etc/fail2ban/jail.d/uptimekuma.local
        owner: root
        group: root
        mode: '0644'
        content: |
          [uptimekuma]
          enabled = true
          # Force polling to ensure we catch file updates ('auto' does not work with uptimekuma)
          backend = polling
          # Read from standard system log where Docker/Rsyslog writes
          logpath = /var/log/syslog

          # Chain must be DOCKER-USER to block traffic before it reaches the container
          chain = DOCKER-USER
          port = 3001
          protocol = tcp

          # Ban Action
          action = iptables-allports[name=uptimekuma, chain=DOCKER-USER]

          # Ban Policy
          maxretry = 4
          findtime = 120
          bantime = 360          
      notify: Restart Fail2Ban

  handlers:
    - name: Restart Fail2Ban
      ansible.builtin.service:
        name: fail2ban
        state: restarted

- name: 3. Clean Up Backup (on AWX)
  hosts: localhost
  connection: local
  gather_facts: no
  become: false

  tasks:
    - name: Remove local backup file from AWX
      ansible.builtin.file:
        path: "/tmp/kuma.db"
        state: absent

Play 1: The "Staging" Phase (Localhost)

  • Goal: Fetch the backup file securely from the cloud to the Ansible Controller (AWX).
  • The challenge: We don't want the remote VPS to have direct access to our Google Cloud bucket credentials. Instead, AWX acts as the secure middleman.
  • Target: localhost (The AWX Execution Environment).
  • Privileges: become: false (We don't need root to download a file).
  • The Tech: We use the google.cloud.gcp_storage_object module.
    • It connects to your bucket (proxmox-backup-bachelor).
    • It authenticates using a Service Account injected securely by AWX env vars.
    • It downloads kuma.db to a temporary location (/tmp/kuma.db) on the AWX runner.

Tip: Check the variables! We define the bucket name and object path as variables at the top so they are easy to change later without breaking the logic.

Play 2a: Kuma Restoration Phase (Remote VPS)

This is where the magic happens on the remote server (galera-witness-hetzner).

  • Goal: Upload the database, launch the container, and set up a "Dead Man's Switch" for the Galera service.

  • The Kuma DB Restore part:

    • Directory Setup: We ensure /opt/uptimekuma exists with correct permissions.
    • The Tech: We use ansible.builtin.copy to move the kuma.db from the AWX controller (local) to the remote VPS.
    • Note: This restores the dashboard(s), notifications and history instantly, although the service needs to be restarted.

  • The Container Launch:

    • Goal: We use the community.docker.docker_container module to spin up Uptime Kuma.
    • Version Control: We explicitly pull louislam/uptime-kuma:beta.
    • Note: If you use v1, then instead of beta, put a 1.

  • Persistence: We mount the host directory (/opt/uptimekuma) to /app/data inside the container.

  • Networking: We expose port 3001 so we can access the dashboard via the VPN tunnel.

  • Time sync: We mount the host's timezone files into the container. This is critical for fail2ban.

Play 2b. The Self-Monitoring Setup (The Push Monitor)

We need this node to scream if the critical Galera Arbitrator (garb) service crashes. Since a Docker container cannot easily see host processes, we use a Push Monitor.

  • Note the Jinja2 Trick - we define the Push URL variable using Jinja2 concatenation:
garb_push_url: "{{ 'http://127.0.0.1:3001/...' + '...' }}"
  • Why? YAML struggles with long lines. This splits the URL cleanly without introducing accidental spaces or newlines that would break the link.
  • The Script (check_garb.sh): ansible writes a bash script directly to /usr/local/bin/.
    • It runs systemctl is-active --quiet garb.
    • If UP: It sends a "heartbeat" (curl request) to the local Uptime Kuma container.
    • If DOWN: It does nothing. Uptime Kuma notices the silence and sends an alert.
  • You might ask why not to do more than just to check for the service being up, such as by running SHOW STATUS LIKE 'wsrep_cluster_size'; to see that the output shows the desired number for connected galera nodes? The challenge is that the garb service does not have the required mariadb-client package.
  • The Cron Job: We schedule this script to run every 2 minutes (*/2) via the root crontab. This ensures we are alerted quickly if the cluster witness goes down.

Play 2c: Set up jail and log monitoring

  • Log Management: Installs rsyslog and configures Docker to send Uptime Kuma logs to the system syslog. This bypasses binary logging issues with journald.
  • Fail2Ban Filter: Creates a custom filter using Regex to identify failed login attempts in the logs, ignoring ANSI color codes sent by the Node.js application.
  • Fail2Ban Jail: Configures a jail that bans offending IPs by manipulating the DOCKER-USER iptables chain, effectively blocking traffic before it even reaches the container.

Play 2d: The High Availability monitoring script

  • Cluster Health Check: Deploys a bash script that uses mariadb-client to connect to Site 1 or Site 2 databases over the VPN.
  • Logic: It queries wsrep_cluster_size. If the size is < 5, it pushes a "Down" signal to Uptime Kuma. If it cannot connect to any node, it reports a connection failure.

Play 3: Clean up on AWX

  • Run a clean up on the localhost (not on the VPS) to remove the previously downloaded kuma.db.
  • This step is optional, it is just to keep things tidy 😇

Create a third job template in AWX

  • Navigate to AWX: Go to Resources -> Templates. Click on the Add -> Add job template.
    • Name: H3 - Restore Uptimekuma from GCP bucket
    • Job Type: Run
    • Inventory: Hetzner (the same inventory used by your other jobs)
    • Project: Select the project containing your playbooks.
    • Playbook: restore-uptimekuma.yml
    • Execution Environment: Your custom EE (with google.cloud and community.docker).
    • Credentials: select your Ansible SSH and GCP UptimeKuma Backup and Witness DB Monitor.
    • Variables: none, as we have the safe-to-share ones already in Gitea
    • Privilege Escalation: tick the box, as play 2 uses the ‘become: true’ parameter.
  • Click Save. Do not launch it just yet, as it will only work once chained with the [1] Provisioning and [2] Configuring job.

Remember to also tick the ‘Privilege Escalation’ tick box that is below the area captured by this screenshot

Troubleshooting the 3rd template

  • Problem 1: Uptimekuma works on http://public_ip:3001 but the Uptimekuma push script is not working.
# Verify the push URL matches the one in your UptimeKuma instance in your VM (owned by root)
sudo nano /usr/local/bin/check_garb.sh

# Verify that the crontab job is present (owner by root)
sudo nano crontab -e

# If the container is in a restart loop and logs show permission errors, 
# ensure the /opt/uptimekuma directory is owned by user 1000 (node):
chown -R 1000:1000 /opt/uptimekuma
  • Problem 2: Unreachable database - if the uptimekuma monitor for checking the size of the cluster fails, try reaching it manually from your VPS - either there is a firewall rule in the way or the credentials are incorrect:
# Firstly, see if you can reach the DB
mariadb -h $HOST -u "$DB_USER" -p"$DB_PASS"
exit;

# Run this from your VPS to ensure connection can be established for y our monitoring user:
mariadb -h $HOST -u "$DB_USER" -p"$DB_PASS" -s -N --connect-timeout=3 -e "SHOW STATUS LIKE 'wsrep_cluster_size';" | awk '{print $2}'
# Fill in the variables above and see if you get the desired output in the form of a number.
  • Problem 3: Uptime Kuma not starting - most likely, a DB corruption issue.
    • Most likely, this is because you re-ran the third job and did not flush the /opt/uptimekuma content after removing the container.
    • If the container is in a restart loop and logs show permission errors, ensure the /opt/uptimekuma directory is owned by user 1000 (node) using chown -R 1000:1000 /opt/uptimekuma.
    • Check the logs:
# Check the uptimekuma logs
sudo docker logs --tail 50 uptimekuma

# Look for lines like these:
2025-11-18T22:30:51Z [DB] INFO: Database Type: sqlite
2025-11-18T22:30:51Z [SERVER] INFO: Connected to the database
2025-11-18T22:31:19Z [DB] ERROR: Database migration failed
2025-11-18T22:31:19Z [SERVER] ERROR: Failed to prepare your database: INSERT INTO "_knex_temp_alter332" SELECT * FROM "stat_hourly"; - SQLITE_CORRUPT: database disk image is malformed

# Fix it by deleting the SQLite files to force a recovery
sudo docker stop uptimekuma

# This is safe to delete when the service is stopped
sudo rm /opt/uptimekuma/kuma.db-wal
sudo rm /opt/uptimekuma/kuma.db-shm

sudo docker start uptimekuma

7. Create a Workflow Template to join the 3 jobs

  • The final state of files in Gitea should look like this:

18 7 create a workflow template

  • We can now join the provisioning and the configuration jobs together. Once the VPS is provisioned in Hetzner, it will then be configured with the S2S VPN details as well as garbd config.
  • In AWX, go to Templates → click on the Add button and choose ‘Add workflow template’ from the drop-down button.
    • Name: H0 - Workflow - Provision + Configure Galera Witness
    • Description: Executed on Hetzner - a VPS is created and set up
    • Inventory: Hetzner

19 7 create a workflow template

  • Then you will be taken to the Visualizer.
    • Add the jobs from H1, H2 to H3.
    • Chain them with the condition for the next step to be triggered only ‘on success’ after the previous job finished.

20 7 create a workflow template

  • If you got back to Templates, it may look something like these four below:

21 7 create a workflow template

  • Then finally, LAUNCH the workflow template!
  • The second template will take the longest time to run (about 3 minutes) because it waits for the first one to finish running the cloud-init template before proceeding further.

22 7 create a workflow template

  • Once the third template runs, you can start checking for the Uptimekuma service on http://public_ip:3001. If you would like to go fancy and have it set up with HTTPS, let me know in the comments and I can update the steps. Upon logging in, if not done already, I would recommend you to set up 2FA, since this service is exposed to the world.
  • Tell me, is it not absolutely beautiful to deploy a VPS with a click of a button and have it set up with all required services and monitoring? Note the reported cluster size in the Message field. Isn’t that neat? Anything smaller than 5 will result in an alert about the cluster being down. With all nodes being up, the cluster size should read 9 (4x on Site 1 + 2x on Site 2 + 1x Witness on Site 3).

23 7 create a workflow template

  • I recommend verifying that fail2ban for Uptime Kuma works as expected. What you can do is to SSH in and run the following commands while trying to login in repeatedly using bogus credentials.
# Watch the logs as you try logging in with bogus credentials:
sudo tail -f /var/log/syslog | grep uptimekuma

# Once you are jailed, you should be able to see it on the status page:
sudo fail2ban-client status uptimekuma

# Unban your public IP address:
sudo fail2ban-client set uptimekuma unbanip 1.2.3.4

This concludes our rather extensive guide. Hopefully there was something new and interesting in there for you to learn without being overwhelming? In case you are missing some required parts in your infrastructure to make it happen, check out my other Tutorials!

Let me know in the comments below how your journey with Ansible and multi-site deployment of Galera in a hybrid infrastructure environment went 😇

More in MariaDB & Galera