2. Create the initial Terraform + Ansible scripts

Step Contents

Firstly, we will create our scripts and based on that, we will create the credential types and add the credentials in. This will hopefully help shedding some light on what we are doing and why.

Scripts required to provision the VPS:
- main.tf : This file is the main set of instructions. It defines the provider, finds the right OS image, and creates the server with your exact specifications. See Terraform manual.
- 1a-provision-witness-terraform.yml – use Terraform to provision the VPS in Hetzner (before cloud-init below is used).
- 1b-provision-witness-hetzner.yml – the cloud-init script that will install and configure services on the VPS. See Hetzner’s manual.
- outputs.tf : This file tells Terraform what information to print out when it’s done. This is critical for AWX integration.

First script – main.tf:

Firewall Management: Creates and attaches a robust Hetzner Cloud Firewall (witness_fw) that strictly limits ingress traffic to SSH, WireGuard, and Uptime Kuma ports (this is external to ufw that also gets installed on the VPS later using the 1b template).

Provider Setup: Configures the hcloud (Hetzner Cloud) provider to manage resources.
Image Selection: Automatically finds the latest available image for Debian 13 (Trixie) on x86 architecture.
Server Creation: Provisions a CX23 VPS (2 vCPU, 4GB RAM) in the Falkenstein data center (fsn1).
Cloud-Init Injection: Injects the 1b-provision-witness-hetzner.yml user data to handle OS-level setup immediately upon boot.

# main.tf

# This tells Terraform we are using the Hetzner Cloud provider
terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1" # Use the latest 1.x version
    }
  }
}

# The provider will automatically use the HCLOUD_TOKEN environment variable
provider "hcloud" {}

# ----------------------------
# --- DEFINE VM PROPERTIES ---
# ----------------------------

# This data block finds the latest "debian-13" image ID
data "hcloud_image" "debian_image" {
  name        = "debian-13"
  with_architecture = "x86"
}

# This data block finds your SSH key to add to the server.
data "hcloud_ssh_key" "jan_key" {
  name = "Jan's key 2025-06"
}
data "hcloud_ssh_key" "ansible_key" {
  name = "Ansible"
}

# This is the main resource block that creates the VM
resource "hcloud_server" "witness_vm" {
  name        = "galera-witness"
  server_type = "cx23"                # 2 vCPU, 4GB RAM, 40GB SSD
  image       = data.hcloud_image.debian_image.id
  location    = "fsn1"                # Falkenstein (eu-central)
  
  # Enable/disable ipv4 and ipv6
  public_net {
    ipv4_enabled = true
    ipv6_enabled = false
  }

  # Add your SSH key for initial access (before cloud-init runs)
  ssh_keys = [
    data.hcloud_ssh_key.jan_key.id,
    data.hcloud_ssh_key.ansible_key.id
  ]

  # This reads the cloud-init for Hetzner and passes it to the server
  user_data = file("1b-provision-witness-hetzner.yml")

  labels = {
    "service" = "galera"
    "role"    = "witness"
  }
}

# -------------------------------------------------
# --- DEFINE THE HETZNER FIREWALL AND ITS RULES ---
# -------------------------------------------------
resource "hcloud_firewall" "witness_fw" {
  name = "galera-witness-fw"

  # Rule 1: Allow SSH (on your new port) from anywhere
  rule {
    direction = "in"
    protocol  = "tcp"
    port      = "2222"
    source_ips = [
      "0.0.0.0/0",
      "::/0"
    ]
  }

  # Allow WireGuard (UDP) from Site 1 and Site 2
  rule {
    direction = "in"
    protocol  = "udp"
    port      = "51821"
    source_ips = [
      "0.0.0.0/0",
      "::/0"
    ]
  }

  # Rule 3: Allow Galera (TCP/UDP) from  VPN subnets
  rule {
    direction = "in"
    protocol  = "tcp"
    port      = "4567"
    source_ips = [
      "192.168.0.0/16",
      "10.10.10.0/24"
    ]
  }
  rule {
    direction = "in"
    protocol  = "udp"
    port      = "4567"
    source_ips = [
      "192.168.0.0/16",
      "10.10.10.0/24"
    ]
  }

  # Rule 4: Allow ICMP (Ping)
  rule {
    direction = "in"
    protocol  = "icmp"
    source_ips = [
      "0.0.0.0/0",
      "::/0"
    ]
  }

  # Rule 5: Allow Uptime Kuma (TCP) from anywhere
  # Later, this can be restricted to the Site 1 + 2 and other WG Roadwarrior IP addresses
  rule {
    direction = "in"
    protocol  = "tcp"
    port      = "3001"
    source_ips = [
      "0.0.0.0/0",
      "::/0"
    ]
  }
}

# -----------------------------------------
# --- ATTACH THE FIREWALL TO THE SERVER ---
# -----------------------------------------
resource "hcloud_firewall_attachment" "fw_attachment" {
  firewall_id = hcloud_firewall.witness_fw.id
  server_ids  = [hcloud_server.witness_vm.id]
}

Terraform provisioning script:

Terraform Execution: Runs terraform apply to provision the actual infrastructure on Hetzner.
Dynamic Inventory: Captures the newly created server’s public IP address from Terraform’s output.
AWX Integration: Automatically adds the new host to the AWX Hetzner inventory, allowing subsequent job templates to target it immediately without manual intervention.
DNS Automation: Updates a CloudFlare DNS A-record (hetzner-witness.bachelor-tech.com) to point to the new IP, ensuring VPN endpoints remain valid even if the IP changes.

# 1a-provision-witness-terraform.yml

---
- name: Provision Hetzner Witness VM with Terraform
  hosts: localhost
  connection: local
  gather_facts: no

  tasks:
    - name: Run Terraform to create the witness server
      community.general.terraform:
        project_path: "{{ playbook_dir }}"
        state: present              # This means "run terraform apply"
        force_init: true            # This runs "terraform init" first
      
      # This is how the playbook gets the Hetzner token
      # from the AWX credential (see step 4)
      environment:
        HCLOUD_TOKEN: "{{ lookup('env', 'HCLOUD_TOKEN') }}"
      
      # This registers the output of the 'terraform apply' command
      register: tf_output

    - name: Show the Witness IPv4 Address
      ansible.builtin.debug:
        msg: "Server '{{ tf_output.outputs.witness_id.value }}' created with IPv4: {{ tf_output.outputs.witness_ipv4.value }}"
    
    - name: Add new VM to AWX Inventory
      awx.awx.host:
        name: "galera-witness-hetzner"
        inventory: "Hetzner"  # Or whatever your inventory is called
        variables:
          ansible_host: "{{ tf_output.outputs.witness_ipv4.value }}"
          ansible_port: 2222
          ansible_user: ansible
        state: present
        
      environment:
        # Token for AWX API - adjust your hostname, as required
        CONTROLLER_HOST: "{{ lookup('env', 'TOWER_HOST') | default('https://awx.bachelor-tech.com', true) }}"
        CONTROLLER_OAUTH_TOKEN: "{{ lookup('env', 'AWX_TOKEN') }}"
        CONTROLLER_VERIFY_SSL: false # Set to true if you have valid SSL

    - name: Update CloudFlare DNS record
      community.general.cloudflare_dns:
         zone: "bachelor-tech.com"
         record: "hetzner-witness"
         type: "A"  # A record is for IPv4
         value: "{{ tf_output.outputs.witness_ipv4.value }}"
         api_token: "{{ cloudflare_api_token }}"
      
      no_log: true # Hides the token from the log output

Cloud-config file

Adds two users (ansible user for management via S2S VPN with AWX from Site 1)
Custom SSH port (with no password auth) + installs packages apart from the one used for Galera
Installs Docker + cofigures ufw firewall rules
The cloud-config file (the file MUST start with the #cloud-config line or else it will not be recognized and the following will not be applied):

# 1b-provision-witness-hetzner.yml

#cloud-config

# Add users
users:
  - name: youruser
    groups: users, admin
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ecdsa-sha2-nistp256 public_key ecdsa-key-20250630
  - name: ansible
    groups: users, admin
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 public_key

# Install the pre-requisites for adding the repo
package_update: true
packages:
  - curl
  - gpg
  # Required for Docker:
  - ca-certificates
  - gnupg
  - python3-pip
  - mariadb-client # To check for Galera cluster size later
package_upgrade: true

# Write into the SSH config file
write_files:
  - path: /etc/ssh/sshd_config.d/ssh-hardening.conf
    content: |
      PermitRootLogin no
      PasswordAuthentication no
      Port 2222
      KbdInteractiveAuthentication no
      ChallengeResponseAuthentication no
      MaxAuthTries 2
      AllowTcpForwarding no
      X11Forwarding no
      AllowAgentForwarding no
      AuthorizedKeysFile .ssh/authorized_keys
      AllowUsers jan ansible      

# Run setup commands
runcmd:
  # Apply the new SSH port
  - systemctl restart sshd
  
  # Manually add the MariaDB repo (from which we will fetch the arbitrator package)
  - curl -o /etc/apt/keyrings/mariadb-keyring.pgp https://mariadb.org/mariadb_release_signing_key.pgp
  # Fetch MariaDB 11.8.5 compatible with Trixie
  - echo "deb [signed-by=/etc/apt/keyrings/mariadb-keyring.pgp] https://deb.mariadb.org/11.8.5/debian trixie main" > /etc/apt/sources.list.d/mariadb.list
  # Update and install the packages
  - apt-get update
  - apt-get install -y fail2ban ufw mc wireguard wireguard-tools rsync galera-arbitrator-4
  
  # Configure them
  - printf "[sshd]\nenabled = true\nport = ssh, 2222\nbanaction = iptables-multiport" > /etc/fail2ban/jail.local
  - systemctl enable fail2ban
  - systemctl start fail2ban

  # --- Install Docker ---
  - install -m 0755 -d /etc/apt/keyrings
  - curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.gpg
  - chmod a+r /etc/apt/keyrings/docker.gpg
  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian trixie stable" > /etc/apt/sources.list.d/docker.list
  - apt-get update
  - apt-get install -y python3-docker docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  # Add ansible user to docker group
  - usermod -aG docker ansible
  - systemctl enable docker
  - systemctl start docker

  # Configure UFW
  - ufw allow 2222/tcp # SSH
  - ufw allow 51821/udp # Site-to-site VPN listening port
  - ufw allow 3001/tcp # UptimeKuma's web interface
  - ufw allow from 192.168.0.0/16 to any port 4567 # Allows the remote LAN to reach the Arbitrator
  - ufw allow from 10.10.10.0/24 to any port 4567 # Site-to-site VPN for Galera Arbitrator
  
  # Enable UFW
  - ufw --force enable

The Outputs file

Exposes Data: Defines exactly which data points (IP addresses, Server ID) Terraform should return to Ansible after the provisioning completes.
Integration Key: This allows the Ansible playbook to read tf_output.outputs.witness_ipv4.value and use it to update DNS and Inventory. The ipv6 address is provided as an optional extra for those who would prefer to use that, instead (in which case, modify the 1a script as well).

# outputs.tf

output "witness_ipv4" {
  description = "The public IPv4 address of the witness server."
  value       = hcloud_server.witness_vm.ipv4_address
}

output "witness_ipv6" {
  description = "The public IPv6 address of the witness server."
  value       = hcloud_server.witness_vm.ipv6_address
}

output "witness_id" {
  description = "The ID of the witness server."
  value       = hcloud_server.witness_vm.id
}

Before we can launch the template, we will need to prepare our environment with regards to credentials (and credential types). This will be slightly tedious but then we can re-use these safely stored variables in multiple templates.

2. Create the initial Terraform + Ansible scripts

First script – main.tf:

Terraform provisioning script:

Cloud-config file

The Outputs file

Leave a Comment Cancel Reply