# How to Deploy Two iRedMail Instances in HA Using a Galera MariaDB Cluster

[TOC]

A single mail server is a single point of failure. If its host, its VM, or its entire site goes offline, every domain it hosts stops receiving mail until someone intervenes. In this tutorial I'll walk through building a second, independent iRedMail node at a second site, so that either node can serve every mailbox, and DNS automatically points at whichever one is actually healthy.

I'll call the two nodes **mail1** (Site 1) and **mail2** (Site 2) throughout. Swap in your own hostnames and IP ranges as you go.

## Why this shape, and not something simpler

A mail server's state splits into three genuinely different problems, and I found it much easier to reason about once I stopped trying to solve them with one tool.

**Accounts, domains, passwords, quotas, DKIM tracking, webmail identities:** all of this lives in SQL. I put it on a Galera multi-master cluster spanning both sites. This is the single most load-bearing decision in this whole design: with Galera there's no "promotion" step, no replica that can lag behind, and no window where two nodes disagree about who's authoritative. A write commits everywhere or nowhere; a node cut off from the cluster simply can't accept writes at all. That's what actually prevents split-brain, not any cleverness on my part.

**Mail content itself**, the message files on disk, doesn't live in SQL. If you're on Dovecot 2.3, its own built-in replication (`dsync`) is the right tool for this and I'd use it without hesitation. If you're on Dovecot 2.4, that option is gone; the project removed replication entirely, and its own upgrade docs simply say *"use NFS or another shared filesystem instead."* I didn't want that either: shared storage across two sites makes the link between them a hard dependency for the passive side to have mail access at all, which defeats the point of having a second site. So I used **Syncthing** to sync the Maildir tree directly between the two nodes. This is safe specifically because the two nodes are **active/passive**, not active/active: only one is ever genuinely "live" at a time, so there's no real concurrent-write race, and Maildir's file-per-message layout with effectively unique filenames means two independently-delivered mailboxes merge safely by simple union.

**Getting traffic to the healthy node** is a third problem neither of the above solves. That's DNS-level failover, covered in Step 10.

```mermaid
flowchart TB
    subgraph Site1["Site 1"]
        M1["mail1<br/>Postfix / Dovecot / Amavis"]
    end
    subgraph Site2["Site 2"]
        M2["mail2<br/>Postfix / Dovecot / Amavis"]
    end
    subgraph Galera["Galera cluster (spans both sites + a 3rd-site arbitrator)"]
        DB[("Accounts, domains, passwords,<br/>DKIM tracking, quota, webmail state")]
    end
    M1 <-->|"reads/writes, synchronous, multi-master"| DB
    M2 <-->|"reads/writes, synchronous, multi-master"| DB
    M1 <-->|"Syncthing: Maildir content,<br/>two-way, over the private tunnel"| M2
```

## Prerequisites

- **A Galera (or Galera-compatible) MariaDB/MySQL cluster already spanning both sites**, with proper quorum protection. Building that cluster is its own project and out of scope here; the short version is that you want synchronous multi-master replication with enough nodes and weighting that losing either site still leaves a majority somewhere, ideally with a lightweight arbitrator (`garbd`) on a third, independent site to break ties. Percona XtraDB Cluster and MySQL InnoDB Cluster both work the same way if you'd rather use those.
- **Two sites with genuinely independent internet connections.** If both sites share one upstream link, there's no real redundancy to build.
- **A reverse proxy at each site capable of host-based routing and NAT port-forwarding** (OPNsense with the HAProxy plugin, Nginx, Traefik, whatever you already run). I'll assume OPNsense/HAProxy below since that's what I use, but the pattern is the same regardless.
- **A DNS provider with health-checked, DNS-only load balancing.** Cloudflare's Load Balancing add-on (from $5/month for two origins) is what I use; any provider that can steer plain DNS answers based on TCP health checks will do. This has to be DNS-only, not proxied through the provider's edge, since raw SMTP/IMAP can't travel through an HTTP-only reverse proxy.
- **A monitoring tool that accepts webhook/push updates** (I use Uptime Kuma; anything similar works).
- **Some comfort with Syncthing.** If you've never used it, its own documentation covers pairing and folder setup well; I'll cover the mail-specific parts here.

## Step 1: Provision the shared Galera databases

iRedMail expects six databases: one for mailboxes and domains, and one each for Amavis, iRedAdmin, RoundCube, iRedAPD, and Fail2ban. Rather than let each node create its own local copies, I create these once, directly on the Galera cluster, and give each service its own scoped user reachable from both sites' subnets.

```sql
CREATE DATABASE IF NOT EXISTS mail_vmail        CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE DATABASE IF NOT EXISTS mail_amavisd      CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE DATABASE IF NOT EXISTS mail_iredadmin    CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE DATABASE IF NOT EXISTS mail_roundcubemail CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE DATABASE IF NOT EXISTS mail_iredapd      CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE DATABASE IF NOT EXISTS mail_fail2ban     CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

-- one pair of users per service: a read-only "bind" user for Postfix/Dovecot
-- lookups, and a full-privilege admin user for iRedAdmin/iRedAPD/etc.
-- Scope each to the two site subnets that actually need access.
CREATE USER 'mail_vmail'@'<site1-subnet>' IDENTIFIED BY '<password>';
CREATE USER 'mail_vmail'@'<site2-subnet>' IDENTIFIED BY '<password>';
GRANT SELECT ON mail_vmail.* TO 'mail_vmail'@'<site1-subnet>';
GRANT SELECT ON mail_vmail.* TO 'mail_vmail'@'<site2-subnet>';

CREATE USER 'mail_vmailadmin'@'<site1-subnet>' IDENTIFIED BY '<password>';
CREATE USER 'mail_vmailadmin'@'<site2-subnet>' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON mail_vmail.* TO 'mail_vmailadmin'@'<site1-subnet>';
GRANT ALL PRIVILEGES ON mail_vmail.* TO 'mail_vmailadmin'@'<site2-subnet>';

-- repeat the same CREATE USER / GRANT pair for mail_amavisd, mail_iredadmin,
-- mail_roundcubemail, mail_iredapd and mail_fail2ban, each with their own
-- dedicated user
```

Run this against any node in the cluster (or a VIP in front of it); Galera's synchronous replication means it's immediately live everywhere.

## Step 2: Install iRedMail on both nodes

iRedMail's own installer doesn't support pointing directly at a remote database; it always sets up a local MariaDB during install. So the pattern for each node is: install normally against a disposable local database, then immediately repoint every component at the shared Galera cluster and turn the local one off for good. Do this identically on both mail1 and mail2.

Install the base packages and download iRedMail:

```bash
sudo apt-get update
sudo apt-get install -y wget tar
mkdir -p /tmp/iredmail-build && cd /tmp/iredmail-build
wget -q https://github.com/iredmail/iRedMail/archive/refs/tags/1.8.4.tar.gz -O iRedMail-1.8.4.tar.gz
tar zxf iRedMail-1.8.4.tar.gz
cd iRedMail-1.8.4
```

Set the node's own hostname (distinct per node; the shared mail identity gets configured separately in Step 3):

```bash
sudo hostnamectl set-hostname mail1.yourdomain.com
```

Build an unattended install config. iRedMail supports a documented unattended path: a plain config file of environment variables, with `AUTO_USE_EXISTING_CONFIG_FILE=y` telling the installer to skip its interactive wizard entirely.

```bash
genpw() { tr -dc "A-Za-z0-9" < /dev/urandom | head -c "${1:-24}"; }

cat > config <<EOF
export WEB_SERVER=NGINX
export BACKEND_ORIG=MARIADB
export BACKEND=MYSQL
export VMAIL_DB_BIND_PASSWD=$(genpw)
export VMAIL_DB_ADMIN_PASSWD=$(genpw)
export MLMMJADMIN_API_AUTH_TOKEN=$(genpw)
export MYSQL_ROOT_PASSWD=$(genpw)
export USE_IREDADMIN=YES
export USE_ROUNDCUBE=YES
export USE_FAIL2BAN=YES
export AMAVISD_DB_PASSWD=$(genpw)
export IREDADMIN_DB_PASSWD=$(genpw)
export RCM_DB_PASSWD=$(genpw)
export IREDAPD_DB_PASSWD=$(genpw)
export FAIL2BAN_DB_PASSWD=$(genpw)
export FIRST_DOMAIN=yourdomain.com
export DOMAIN_ADMIN_PASSWD_PLAIN=$(genpw 20)
EOF
chmod 600 config
```

Run the installer. We answer "no" to iRedMail managing its own firewall rules, since our reverse proxy is already the perimeter, and run it detached so a dropped SSH session doesn't kill a fifteen-minute install:

```bash
sudo bash -c "
cd /tmp/iredmail-build/iRedMail-1.8.4
nohup env \
  AUTO_USE_EXISTING_CONFIG_FILE=y \
  AUTO_INSTALL_WITHOUT_CONFIRM=y \
  AUTO_CLEANUP_REMOVE_SENDMAIL=y \
  AUTO_CLEANUP_REPLACE_FIREWALL_RULES=n \
  AUTO_CLEANUP_RESTART_FIREWALL=n \
  AUTO_CLEANUP_REPLACE_MYSQL_CONFIG=y \
  DEBIAN_FRONTEND=noninteractive \
  bash iRedMail.sh > /tmp/iredmail-install.log 2>&1 < /dev/null &
disown
"
```

Watch for the real completion marker, not just "the process is still running":

```bash
tail -f /tmp/iredmail-install.log | grep -m1 "Congratulations\|fatal\|ERROR"
```

Then verify the evidence, not the banner:

```bash
sudo postconf -n | grep -i "PH_\|fatal" || echo clean
sudo systemctl is-active postfix dovecot mariadb nginx amavis clamav-daemon fail2ban php8.4-fpm iredapd
sudo systemctl --failed
sudo reboot
```

A reboot is worth doing once, to confirm everything comes up cleanly on its own rather than relying on install-time startup order.

## Step 3: Point every component at the shared cluster

Now repoint each iRedMail component from its disposable local database to the shared one from Step 1. This table is worth keeping open while you edit:

| Component | What it does | Config file |
|---|---|---|
| Postfix | The mail transfer agent: accepts, routes and delivers SMTP mail | `/etc/postfix/mysql/*.cf` (all files) |
| Dovecot | The IMAP/POP3 server: stores and serves mailbox content to mail clients | `/etc/dovecot/dovecot.conf` |
| Amavis | Sits between Postfix and the content filters (ClamAV, SpamAssassin), scanning and DKIM-signing mail | `/etc/amavis/conf.d/50-user` |
| RoundCube | The webmail interface end users log into | `config/config.inc.php` |
| iRedAdmin | The admin panel for managing domains, mailboxes and aliases | `settings.py` |
| iRedAPD | Postfix policy daemon: greylisting, throttling, sender/recipient checks | `settings.py` |
| Fail2ban | Watches auth logs and temporarily bans IPs showing brute-force patterns | `/root/.my.cnf-fail2ban` |

Each of these needs its host changed from `127.0.0.1` to the Galera cluster's address, and its database name and credentials changed from the local `vmail`/`amavisd`/etc. to the shared `mail_vmail`/`mail_amavisd`/etc. from Step 1.

> **One Dovecot-specific detail worth knowing before you start:** on Dovecot 2.4, the entire effective configuration lives in one flat `/etc/dovecot/dovecot.conf`; the familiar `conf.d/*.conf` layout iRedMail's installer still generates is not actually read at all. The live database connection is an inline block directly inside that file:
> ```
> mysql 127.0.0.1 {
>     port = 3306
>     dbname = vmail
>     user = vmailadmin
>     password = ...
> }
> ```
> Edit this block in place (changing the host, `dbname`, `user` and `password`), not any file under `conf.d/`. The same applies to Dovecot's `hostname` setting if you set one, and to anything else Dovecot-related: if it isn't in `dovecot.conf` itself, Dovecot 2.4 isn't reading it.

After editing every file in the table, restart the affected services and confirm each is reading from Galera, not local, with a real end-to-end test:

```bash
sudo systemctl restart postfix dovecot amavis php8.4-fpm nginx iredapd fail2ban

printf "Subject: Galera backend test\n\nTest.\n" | sudo sendmail -f postmaster@yourdomain.com postmaster@yourdomain.com
sleep 5
sudo doveadm mailbox status -u postmaster@yourdomain.com messages INBOX
```

Once every component is confirmed reading from Galera, on both nodes, stop and disable the local MariaDB entirely rather than leaving it running unused:

```bash
sudo systemctl stop mariadb
sudo systemctl disable mariadb
```

Re-run the mail delivery test one more time with local MariaDB fully gone, to prove nothing was still quietly depending on it.

## Step 4: Give each node its own DKIM identity

DKIM keys don't need to be shared between nodes. Each node gets its **own keypair per domain**, published under its **own selector**, so there's never anything to keep in sync:

```bash
sudo amavisd genrsa /var/lib/dkim/yourdomain.com.pem 2048
sudo chown amavis:amavis /var/lib/dkim/yourdomain.com.pem
sudo chmod 600 /var/lib/dkim/yourdomain.com.pem
```

Register it in Amavis, using a selector name that identifies the node:

```
dkim_key('yourdomain.com', 'mail1-dkim', '/var/lib/dkim/yourdomain.com.pem');   # on mail1
dkim_key('yourdomain.com', 'mail2-dkim', '/var/lib/dkim/yourdomain.com.pem');   # on mail2
```

Get the DNS value and publish both selectors, for every domain:

```bash
sudo amavisd showkeys
```

```
mail1-dkim._domainkey.yourdomain.com.  TXT  "v=DKIM1; p=..."
mail2-dkim._domainkey.yourdomain.com.  TXT  "v=DKIM1; p=..."
```

Whichever node actually signs an outgoing message, its own selector is already live in DNS. If you're renaming a selector on an already-live node, publish the new DNS record first and flip the signing config only afterward, so there's never a window where a node signs with a selector that doesn't resolve yet.

## Step 5: Migrating an existing single-node install into this setup

If you already have one mail server running its own local database (the common iRedMail default) and want to bring it into this HA design, the work is almost identical to Steps 2 and 3, just with one extra step first: dump your existing data and import it into the new Galera-hosted databases before repointing, so you don't lose your existing domains, mailboxes and DKIM history.

```bash
mysqldump --single-transaction --routines --skip-triggers --no-create-db vmail \
  | mysql -h <galera-address> -u mail_vmailadmin -p mail_vmail
# repeat for amavisd, iredadmin, roundcubemail, iredapd, fail2ban
```

`--skip-triggers` is there because the scoped users deliberately don't have `SUPER`, which one schema trigger (auto-populating a `domain` column on insert into `used_quota`) needs to be created. Recreate that one trigger by hand, once, with full admin credentials:

```sql
DELIMITER $$
CREATE TRIGGER used_quota_before_insert
    BEFORE INSERT ON used_quota FOR EACH ROW
    BEGIN
        SET NEW.domain = SUBSTRING_INDEX(NEW.username, '@', -1);
    END$$
DELIMITER ;
```

From there, follow the same table in Step 3 to repoint every component, give the node its own DKIM selector as in Step 4, and stop the local database once everything checks out. Your second node then follows the plain fresh-install path in Steps 2 to 4, since it starts empty and just needs to see the same shared data your first node already migrated in.

## Step 6: Sync mailbox content with Syncthing

Install Syncthing on both nodes from the official repository, running as the same system user that already owns the Maildir tree (`vmail` on a standard iRedMail install):

```bash
sudo mkdir -p /etc/apt/keyrings
sudo curl -sL -o /etc/apt/keyrings/syncthing-archive-keyring.gpg https://syncthing.net/release-key.gpg
echo "deb [signed-by=/etc/apt/keyrings/syncthing-archive-keyring.gpg] https://apt.syncthing.net/ syncthing stable-v2" \
  | sudo tee /etc/apt/sources.list.d/syncthing.list
sudo apt-get update && sudo apt-get install -y syncthing jq
```

Point it at a dedicated config directory with a systemd override, and start it as `vmail`:

```ini
# /etc/systemd/system/syncthing@.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/syncthing serve --home="/opt/syncthing-config" --no-browser --no-restart
```

```bash
sudo mkdir -p /opt/syncthing-config && sudo chown vmail:vmail /opt/syncthing-config
sudo systemctl daemon-reload
sudo systemctl enable --now syncthing@vmail.service
```

Pair the two nodes and share one folder, using Syncthing's own `cli` rather than hand-editing its config:

```bash
# on mail1, using mail2's device ID (find it via `syncthing cli --home /opt/syncthing-config show system`)
syncthing cli --home /opt/syncthing-config config devices add \
  --device-id <mail2-device-id> --name mail2 --addresses tcp://<mail2-address>:22000

syncthing cli --home /opt/syncthing-config config folders add \
  --id vmail-ha --label "vmail HA sync" --path /var/vmail --type sendreceive

syncthing cli --home /opt/syncthing-config config folders vmail-ha devices add \
  --device-id <mail2-device-id>

# repeat symmetrically on mail2, using mail1's device ID
```

Scope the sync to actual mail content, not the whole directory. `/var/vmail` also holds each node's own local SQL backup dump and a couple of unused mailing-list directories, none of which should be merged between nodes, plus Dovecot's own local performance caches, which are unsafe to sync mid-write and rebuild automatically if missing. Exclude all of it with a `.stignore` file at the folder root:

```
# host-local, not mail content -- never sync
/backup
/mlmmj
/mlmmj-archive
/pgp-keys

# Dovecot index/cache files: local performance cache only, rebuilt
# automatically if missing. dovecot-uidlist is NOT excluded; it encodes
# IMAP UID assignments and should stay consistent across replicas.
(?d)dovecot.index
(?d)dovecot.index.log*
(?d)dovecot.index.cache*
(?d)dovecot.list.index*
```

```bash
sudo touch /var/vmail/.stignore && sudo chown vmail:vmail /var/vmail/.stignore
```

Turn on trash-can versioning, so an accidental deletion propagating between nodes isn't unrecoverable:

```bash
syncthing cli --home /opt/syncthing-config config folders vmail-ha versioning type set trashcan
syncthing cli --home /opt/syncthing-config config folders vmail-ha versioning params set cleanoutDays 14
```

Syncthing needs to create a couple of marker files at the folder root itself (`.stfolder`, confirming the folder genuinely exists; `.stversions`, for the trash can). Since `/var/vmail`'s own top-level directory typically isn't writable by the `vmail` user (only its subdirectories are), pre-create both as root and hand ownership over:

```bash
sudo touch /var/vmail/.stfolder && sudo chown vmail:vmail /var/vmail/.stfolder
sudo mkdir -p /var/vmail/.stversions && sudo chown vmail:vmail /var/vmail/.stversions
sudo systemctl restart syncthing@vmail.service
```

Give it a few minutes, then confirm both sides report the same file count and an idle, error-free state:

```bash
API_KEY=$(sudo grep -oP '(?<=<apikey>)[^<]+' /opt/syncthing-config/config.xml)
curl -s "http://127.0.0.1:8384/rest/db/status?folder=vmail-ha" -H "X-API-Key: $API_KEY" \
  | grep -E '"state"|"error"|globalFiles|localFiles|needFiles'
```

If both nodes have ever independently run the iRedMail installer, the very first sync can hit a **filename collision**: the installer sends a fixed-name postmaster welcome email (`mua.eml`, `links.eml`, and so on) on every install, and those aren't organically-delivered mail with unique names the way real messages are. Syncthing handles this correctly on its own, keeping both copies and renaming the loser to `*.sync-conflict-<timestamp>-<device>` rather than silently discarding anything; it's a one-time thing to tidy up, not an ongoing concern.

## Step 7: Keep inter-node traffic private, and harden Fail2ban

If your two sites already have a site-to-site VPN between them (WireGuard, IPsec, OpenVPN, whatever), route Galera replication and Syncthing sync over it rather than the public internet. This keeps your two nodes' own housekeeping traffic off the public internet entirely, avoids exposing MySQL and Syncthing ports on either public IP, and, importantly, gives Fail2ban something stable to trust.

That last point matters more than it looks. Testing or monitoring that reaches a node via its **public** IP (which is exactly what a real inbound sender does) can trip protections like Fail2ban's `pregreet` filter, which watches for clients that violate SMTP protocol before the server's own greeting. Since Fail2ban's default `ignoreip` already exempts private RFC1918 ranges, this only bites cross-site traffic that happens to be flowing over the public internet.

The fix isn't to whitelist each site's public IP; that would blind Fail2ban to exactly the kind of traffic it exists to police, since real inbound mail legitimately arrives from the public internet on both sides too. Instead, whitelist the **VPN tunnel's own private address range**, which only ever carries your own infrastructure traffic:

```
# on mail1's fail2ban, add mail2's tunnel address (or subnet)
ignoreip = 127.0.0.1 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 <mail2-tunnel-address>

# on mail2's fail2ban, add mail1's tunnel address (or subnet)
ignoreip = 127.0.0.1 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 <mail1-tunnel-address>
```

If either site sits behind a dynamic secondary WAN, point the VPN's peer/endpoint setting at that site's dynamic DNS hostname rather than a hard-coded IP. Most VPN implementations, including WireGuard, re-resolve the endpoint hostname automatically on a failed handshake, so the tunnel reconnects on its own if a site fails over to its backup connection, without any manual intervention.

Worth knowing for the failure case: if a site goes down entirely, its side of the tunnel goes down too, and that's fine, since there's no node left there to sync with anyway. The actual mail failover in Step 10 never depends on this tunnel at all; real senders reach the surviving site over the public internet directly. The tunnel exists purely for the two nodes' own private traffic while both sites are up. One edge case worth being aware of: if the tunnel itself drops while both sites are individually still healthy, that looks like a network partition to Galera too, which is exactly why a quorum arbitrator on a third, independent site (mentioned in the prerequisites) matters, so the cluster can still resolve correctly even when it's specifically the link between your two mail sites that's broken.

## Step 8: Automatic failover with a DNS-only Load Balancer

Two things worth checking before building this:

1. **Is your mail hostname actually attached to a Load Balancer, or is it a plain static DNS record?** It's easy for it to end up as the latter, especially if a Load Balancer already protects your main website and you assume mail rides along with it. Nothing fails over automatically until the mail hostname itself is under LB management.
2. **A proxied Load Balancer is HTTP(S)-only.** It can't carry raw SMTP or IMAP through the provider's edge. The Load Balancer covering your mail hostname needs to be **DNS-only** (unproxied): it decides which IP address to answer with, based on health checks, and the actual mail connection goes straight to whichever site's reverse proxy, exactly the way your NAT and HAProxy setup already expects.

```mermaid
sequenceDiagram
    participant Sender as Remote MTA / mail client
    participant LB as DNS-only Load Balancer
    participant R1 as Reverse proxy, Site 1
    participant R2 as Reverse proxy, Site 2
    Sender->>LB: resolve mail.yourdomain.com
    LB-->>Sender: A record (whichever pool is healthy)
    alt Site 1 healthy
        Sender->>R1: SMTP/IMAP direct, NAT to mail1
    else Site 1 down, Site 2 takes over
        Sender->>R2: SMTP/IMAP direct, NAT to mail2
    end
```

If you already have Load Balancer pools set up for your website, reuse them for mail rather than creating new ones; most providers, Cloudflare included, bill per origin server, not per Load Balancer, and attaching a second Load Balancer object to already-existing pools costs nothing extra. The trade-off is that mail's failover then rides on the website's own health check (typically HTTP) rather than a mail-specific one (say, a TCP check on port 993). Given that a real site outage takes the website and the mail server down together, that trade-off is often perfectly reasonable; build a dedicated mail-specific health check and pools if you want tighter detection and don't mind the extra cost.

With Cloudflare, creating the mail-specific Load Balancer looks like this:

```bash
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE/load_balancers" \
  -H "X-Auth-Email: $CF_EMAIL" -H "X-Auth-Key: $CF_KEY" -H "Content-Type: application/json" \
  --data '{
    "name": "mail.yourdomain.com",
    "proxied": false,
    "enabled": true,
    "steering_policy": "off",
    "default_pools": ["<site1-pool-id>"],
    "fallback_pool": "<site2-pool-id>",
    "ttl": 60
  }'
```

`"steering_policy": "off"` means strict priority failover: use the default pool until it's genuinely unhealthy, then fall back, rather than distributing traffic across both. For mail, a clear "who's in charge right now" answer is what you want, not load balancing.

## Step 9: Monitor every layer

Each layer built above deserves its own check, since a failure in any one of them can look identical to "everything's fine" from the others' point of view.

**Galera cluster health.** If you already monitor the cluster for other services, this is likely already covered; otherwise, a monitor that checks `wsrep_cluster_status` and `wsrep_local_state_comment` on each node catches a partition or a node falling out of sync.

**Syncthing folder state.** Deploy a small monitor script on each node that polls Syncthing's local REST API for every folder's status and pushes a consolidated result to your monitoring tool, checking hourly when healthy and retrying every minute when something's wrong:

```bash
SYNCTHING_API_KEY="..."   # from each node's own Syncthing config
UPTIME_KUMA_PUSH_URL="http://<your-monitoring-host>/api/push/<token>"
```

Run one instance per node, each pushing to its own push token; sharing one push URL between both nodes conflates their statuses into a single indistinguishable signal, exactly the ambiguity this exists to eliminate.

**The actual mail-facing surface**, which is what your users and remote senders experience directly:

- A TCP or TLS-certificate monitor against port 465 (SMTPS) on each node, confirming the listener is up and the certificate isn't expired.
- An HTTP(S) monitor against the webmail login page, expecting a `200` response.
- An HTTP(S) monitor against the admin panel, same expectation.
- Nginx itself is effectively covered by the two checks above; if you want it monitored independently, a stub-status endpoint or a simple TCP check on its listening port works.

*(screenshot: monitor list showing all checks green for both nodes)*

*(screenshot: one monitor's detail view, showing its response-time history)*

## Step 10: Test the failover, and troubleshoot if mail doesn't arrive

Test by forcing the primary pool unhealthy in your Load Balancer (the same state a real health-check failure would trigger), and confirm, in order:

1. **DNS actually flips**, within your configured TTL, when queried from a public resolver.
2. **Mail actually flows through the newly-active node**, reached via its public IP directly (bypassing any local DNS cache): a real TLS handshake on port 465/993 presenting the correct certificate, and a genuine SMTP banner on port 25.
3. **Webmail and the admin panel both load from the new node.**
4. **A real login succeeds**, using an account whose password you actually know, not just a green checkmark somewhere. This is the step that catches the most, since a config-only check can look perfectly healthy while an actual login silently fails.

Revert once satisfied, and confirm DNS flips back within the TTL too.

If mail genuinely isn't getting through during testing (or for real), work through the layers in order rather than guessing:

**Is the message reaching Postfix at all, and what happened to it?**
```bash
sudo tail -f /var/log/mail.log
```
Look for `connect from`, `reject`, `status=bounced`, or `status=deferred` lines around the time in question. A missing `connect from` entry for an external sender's IP, with no corresponding bounce visible anywhere, is the specific signature of a NAT or firewall rule not actually forwarding the connection; check both halves of your port-forward configuration (the NAT rule and its paired firewall pass rule), since editing one without the other is an easy, very quiet mistake.

**Did the content filter or DKIM signing choke on it?**
```bash
sudo grep amavis /var/log/mail.log | tail -30
```

**Did authentication or final delivery fail?**
```bash
sudo journalctl -u dovecot -f
```
`doveadm auth test <user>` exercises the exact same SQL path a real login uses, without needing a mail client:
```bash
echo 'the-password' | sudo doveadm auth test someone@yourdomain.com
```

**Is the sender's IP (or your own testing IP) actually banned?**
```bash
sudo fail2ban-client status
sudo fail2ban-client status pregreet
```

**Is the node actually talking to Galera, and does the account exist there?**
```bash
mysql -h <galera-address> -u mail_vmailadmin -p -e "SELECT username, active FROM mailbox WHERE username='someone@yourdomain.com';" mail_vmail
```

**Is the mail content actually synced to the node currently receiving traffic?** Check Syncthing's folder status via its API or GUI (Step 6); a folder stuck in an error state, or simply paused, means the currently-active node may be missing recent content the other node has.

**Is DNS actually resolving to the node you expect right now?**
```bash
dig +short mail.yourdomain.com @1.1.1.1
```

Working through these in order, from "did the packet even arrive" outward to "did the application logic accept it", finds the actual break far faster than jumping straight to the layer that seems most likely.