1. Proposed architecture + infrastructure requirements

AWX will run the following three templates as part of one workflow:

1. Provision a VPS on Hetzner using Terraform inc. fail2ban, ufw and Hetzner-based external firewall rules.
2. Configure a site-to-site VPN (on WireGuard) and garb for Galera Arbitrator monitoring.
3. Recover a backup of Uptime Kuma in a docker environment that monitors the health of the cluster and any websites or other services of your choice.

With a click of a button, you will be able to spin up a fully configured VPS for < 5 EUR a month in an environment of a trusted european-based cloud provider and be able to re-create it whenever or wherever needed 🙂

Use case – what is this setup for?

• You have Site 1 and Site 2 already configured using a MariaDB Galera Cluster and web servers.
• Site 1 and Site 2 are connected using a Site-to-site VPN tunnel (WireGuard), each with equal weight in terms of Galera weighting.
• Site 3 will therefore act as a ‘witness’ in between the two sites – if one goes down, the DB cluster will remain operational.

Proposed architecture:

• 1st Site/Segment 1 (U vody) – main site:
  • Proxmox host 1
    • Galera-A1 LXC – Weight – 1
    • Galera-A2 LXC – Weight – 1
    • Web1 VM
    • OPNSense 1 (CARP – master, HAProxy, S2S VPN)
  • Proxmox host 2
    • Galera-A3 – Weight – 1 (Proxmox host 2)
    • Galera-A4 – Weight – 1 (Proxmox host 2)
    • Web2 VM
    • OPNSense 2 (CARP – backup, HAProxy, S2S VPN)
• 2nd Site/Segment 2 (Tusarka) – fallback site:
  • Proxmox host 3
    • Galera-A5 – Weight 2 (Proxmox host 3)
    • Galera-A6 – Weight 2 (Proxmox host 3)
    • Web3 VM
    • OPNSense 3 (HAProxy, S2S VPN)
• 3rd Site/Segment 3 (Hetzner) – witness & backup site:
  • Galera witness – Weight – 1 (S2S VPN)
  • GCP or Hetzner Bucket – regular off-site backups (PBS + scripts + Uptimekuma SQLite database for restoration during re-provisioning)

Galera set up on Site 1 & 2 – weights & segments

While the Site 1 & 2 configuration is out of the scop of this tutorial, we have already explored which firewall ports will need to be opened and how to troubleshoot them (at least on OPNSense).

• In case you have been running your Galera cluster on just one site for now, you may have noticed that the config file for the Arbitrator contains a term ‘segment’. What are they?
  • Galera segments allow you to group nodes based on their physical location (e.g., Site 1, Site 2, Cloud). By assigning these segments, you enable Optimized WAN Replication: instead of the primary node sending individual data packets to every remote node over your VPN tunnel (which consumes massive bandwidth), it sends a single copy to one node in the remote segment, which then acts as a relay to distribute the data locally to its neighbors. This drastically reduces traffic across your site-to-site links and prevents database replication from choking your network.
• In its default setting, each node carries one weight. In the proposed architecture section earlier, you may have noticed that Site 2 has only 2x Galera nodes, whereas Site 1 has 4x nodes. So how can you influence weighting on the galera nodes?
• An example for Site 2’s Galera node config where we need to set:
  • segment: 2
  • weight for each node: 2

sudo nano /etc/mysql/mariadb.conf.d/60-galera.cnf

[galera]
# ... your other settings ...

# Site 2 Specifics: Segment 2, Weight 2
wsrep_provider_options="gcache.size=512M;gcs.fc_limit=128;gcs.fc_factor=0.8;gmcast.segment=2;pc.weight=2"

# Full Cluster List
wsrep_cluster_address="gcomm://192.168.8.71,192.168.8.72,192.168.8.73,192.168.8.74,192.168.6.75,192.168.6.76"

# Node Specifics (Example for A5)
wsrep_node_address = "192.168.6.75"
wsrep_node_name = "galera-a5"
wsrep_sst_receive_address = "192.168.6.75"

• Similarly, on Site 1, ensure you use:
  • weight: 1 (since we have 4 nodes)
  • segment: 1 (as our first site)

[galera]
# ... your other config ...
wsrep_provider_options="gcache.size=512M;gcs.fc_limit=128;gcs.fc_factor=0.8;gmcast.segment=1;pc.weight=1"

• Once se tup, SSH into any Galera node in your cluster and compare the values:

mysql -u root -p

# With 4+4+1 design, you should see 9
SHOW STATUS LIKE 'wsrep_cluster_weight';

# With 4 nodes + 2 nodes + 1 witness, you should see 7
SHOW STATUS LIKE 'wsrep_cluster_size';

Other Infrastructure Pre-requisites:

• These you can have running on either of your Sites as VMs or containers.
• A Gitea server (or some other source versource version control service hosted locally or externally).
• AWX (or plain Ansible if you are comfortable running playbooks manually). What will you need for the execution environment (keep in mind that this means both the modules in requirements.yml as well as related dependencies in requirements.txt – check our this part of my previous guide).
  • hetzner.hcloud – to provision the VPS and firewall rules on Hetzner
  • community.general – for CloudFlare DNS record changes, Terraform
  • ansible.netcommon – sudo elevation on the VPS
  • awx.awx – to add the VPS into our AWX inventory
  • google.cloud: to interface with a GCP bucket
  • community.docker: to work with a Docker container for UptimeKuma
• An account in Hetzner – we will be deploying a VM for 5 EUR a month
• Reverse proxy and load balancer on Site 1 + 2 (I use OPNSense in this tutorial)
• WireGuard Site-to-Site VPN set up on Site 1 + 2 already
• CloudFlare account – to automate managing DNS records for Site 3 as the public IP of the instance may change during re-provisioning.
• For the third template, I will have a kuma.db (backup) of Uptime Kuma to restore from a GCP bucket.

Now let’s create our first Terraform and Ansible scripts to make this happen!

Go back to course overview: Set up a Galera Witness on Hetzner VPS using Terraform + Ansible (AWX)