# Deploy Vaultwarden in multi-site environment in HA (Docker, OPNSense, Galera cluster, Nginx) [TOC] ## Introduction Running a mission-critical website on a single server creates a significant risk, as routine maintenance or unexpected hardware failures can take your service offline instantly. This tutorial guides you through transforming a standalone web server into a resilient High Availability (HA) cluster using accessible open-source tools. We will utilize **Proxmox** to clone your existing environment and deploy **OPNSense** with **HAProxy** to intelligently distribute traffic between multiple nodes. To handle dynamic content, we solve the challenge of file synchronization by implementing **Syncthing**, a decentralized peer-to-peer tool that keeps folders like WordPress uploads identical across servers in real-time. We will configure **Syncthing** on headless Linux instances, tunnel securely to its GUI for management, and set up automated health monitoring using **UptimeKuma**. By the end, you will have a robust architecture that can survive individual server reboots without downtime, offering a simpler alternative to complex enterprise-grade distributed file systems. ## 1. Vaultwarden or Bitwarden? - While **Bitwarden **offers a robust, enterprise-ready self-hosted option, it is resource-intensive. The official Bitwarden server relies on a complex stack of MSSQL (Microsoft SQL Server) and multiple .NET containers, often requiring 2GB+ of RAM just to idle. - **Vaultwarden** is a lightweight rewrite of the Bitwarden API in Rust. It is fully compatible with official Bitwarden apps (browser extensions, mobile, desktop) but runs on a fraction of the resources (often <100MB RAM). - Bitwarden offers a free option for individuals. If you have a family or an organization and want to share a collection of passwords, you would need to go on the paid tier. The advantages include convenience (no maintenance on your part) and no missing features. - The obvious pitfall with Vaultwarden is that if you deploy it yourself, you will also need to handle the maintenance. We counter that by using docker containers that can be updated very easily. What takes more effort is to ensure your web and database clusters are up to date and secure as well, which is beyond the scope of this tutorial. In addition, if Bitwarden introduces some flashy new features, you may need to wait till the Rust devs update them for Vaultwarden. ### Pre-requisites - A web server (or two) on site 1 + 2 running nginx on Debian 12 or newer. Their set up is beyond the scope of this article - you can refer to my previous article on [**How to configure High Availability for a Web Server using Syncthing and HAProxy (on OPNSense)**](https://bachelor-tech.com/tutorials/how-to-create-high-availability-for-a-web-server-using-syncthing-and-haproxy/). - A Galera cluster (MySQL) deployed in HA on your two sites, ideally with a Galera witness on site 3 (all connected via WireGuard). Again, the set up is beyond the scope of this article - you can complete them using previous tutorials titled [**Deploy MariaDB Galera Cluster on Proxmox**](https://bachelor-tech.com/detailed-guides/deploy-mariadb-galera-cluster-on-proxmox/) and [**Set up a Galera Witness on Hetzner VPS using Terraform + Ansible (AWX)**](https://bachelor-tech.com/tutorials/set-up-galera-witness-on-hetzner-using-terraform-ansible-awx/) - OPNSense (or pfsense or similar) with an HAProxy module to act as a reverse proxy + virtual IP interface for our DB cluster on site 1 and 2. Previously, this was completed in a tutorial called [**OPNSense in HA with CARP with dual WANs**](https://bachelor-tech.com/detailed-guides/opnsense-in-ha-with-carp-with-dual-wans/). ### Topology - **Site 1 (Main)** - all connected to a UPS managed from an RPI - OPNSense in HA (192.168.8.254) - CARP 0 - see **this guide to set it up**. - `OPNSense1` (192.168.8.1) - Proxmox host 1 - `OPNSense2` (192.168.8.2) - Proxmox host 2 - Site-to-site VPN WireGuard) with the interface of 10.10.10.1/24 - Web servers in HA - reachable via a shared back-end pool in HAProxy within OPNSense - `web1` (192.168.8.9) - Proxmox host 1 - Syncthing from web1 to web2 & web1 to web3 - to sync user data in near real-time - Connected to a local Gitea server - to update app data on demand - `web2` (192.168.8.10) - Proxmox host 2 - Syncthing from web2 to web1 & web2 to web3 - to sync user data in near real-time - Connected to a local Gitea server - to update app data on demand - Database Cluster - MariaDB Galera cluster - reachable on virtual IP 192.168.8.70 configured in OPNSense - `galera1-2` (192.168.8.71 + 72) - Proxmox host 1 - `galera3-4` (192.168.8.73 + 74) - Proxmox host 2 - `galera-template` - a pre-prepared LXC template ready for easy deployment in case a replacement is needed (can be automated with AWX for deployment) - `Gitea LXC` (192.168.8.21) - to sync app data and deployment scripts + config.xml for each OPNSense host. - `Uptimekuma1` (192.168.8.60) - to monitor all Site 1 hosts’ uptime inc. services like Syncthing - `RPI` (192.168.8.16) - Corosync for HA for the Proxmox cluster) - ensure a host is reachable if one is down - Proxmox Backup Server with an added drive - for Site 1 + 2. - `APCupsd` with scripts for smooth graceful shutdown of Proxmox hosts - see **this tutorial **for more information. - Can act as a local Galera witness [before Site 3 is set up](https://bachelor-tech.com/tutorials/set-up-galera-witness-on-hetzner-using-terraform-ansible-awx/). - **Site 2 (Online Backup)** - `OPNSense3` (192.168.6.1) - Proxmox host 3 - Site-to-site VPN (WireGuard) with the interface of 10.10.10.2/24 - `Web3` (192.168.6.10) - Proxmox host 3 - Syncthing from web3 to web1 & web3 to web2 - to sync user data in near real-time - Connected to Site 1’s Gitea server - to update app data on demand - `Galera4+5` (192.168.6.75 + 76) - Proxmox host 3 - `Uptimekuma2` (192.168.6.60) - monitors Site 2 services inc. web2’s Syncthing jobs. - **Site 3 (Witness)** - Galera witness VPS deployed automatically via AWX in Hetzner - see [**this tutorial**](https://bachelor-tech.com/tutorials/set-up-galera-witness-on-hetzner-using-terraform-ansible-awx/) on how to achieve that. - Site-to-site VPN (WireGuard) with the interface of 10.10.10.3/24 - `Garb` - daemon that does not hold data and provides HA if one site is down (quorum voting as +1) - `Uptimekuma3` - monitors uptime for websites that are provided by either site (such as bachelor-tech.com) ### Software Versions at the time of write-up For the purpose of reproducibility, as a side note, you can find the versions that I worked with: - OPNSense: - Core Version: 25.7.10 (commit: c2f076f30) - os-haproxy plugin: 4.6_1 - os-acme plugin: 4.11 - os-ddclient plugin: 1.28 - os-git-backup: 1.1_1 - Vaultwarden Docker Image: [1.35.2](https://hub.docker.com/r/vaultwarden/server) - Web servers - OS: Debian 12 (Bookworm, kernel 6.1.0-26-amd64) - nginx: 1.28.0 - Docker Engine (client + server): 29.1.3 - Syncthing: 2.0.12 (Hafnium Hornet) - Fail2ban (server + client): 1.0.2 - Galera cluster: - OS: Debian 13 (Trixie, kernel 6.17.2-2-pve) - Maria DB Server: 11.8.3 - Maria DB Client: 15.2 - UptimeKuma: 2.0.0-beta.4 ### Gotchas with Vaultwarden & Syncthing There are two specific limitations to be aware of during an Active-Active setup for Vaultwarden: - **Token Signing Keys:** Users will get logged out if they hit `web1` but their token was signed by `web2` or `web3` using a different key. You **must** ensure the RSA key files in the data directory are identical across all nodes. - We will handle that in this guide by using Syncthing to distribute the RSA key between the web nodes. - **WebSockets:** Vaultwarden does not currently have a "message bus" (like Redis) to broadcast WebSocket events between nodes. If a user updates a password on `web1`, a device connected to `web2` or `web3` won't get the live update immediately. It will sync the next time the user manually refreshes or performs an action. This is a minor inconvenience but acceptable for most. - Vaultwarden tables utilize Primary Keys, which is good (Galera requires them). However, ensure your Galera nodes are not running in `ENFORCING` mode for `wsrep_drupal_282555_workaround` or strict checking that might block `INSERT` statements if they momentarily lack a PK. Most likely, this will not be an issue. --- ## 2. Create a Vaultwarden DB + Install Dependencies Now since the topology and the specifics of Vaultwarden have been covered, let us move forward with the installation and setup. We will be setting it up on Debian 13 (Trixie) - other versions will likely work in a very similar fashion. ### Create a DB on your cluster - SSH into any of the Galera cluster node (as they will sync) and run the following: - Replace the ‘`your_secure_password`’ string with your own! It is recommended to avoid using special characters for compatibility - the length is what matters the most here! If you do use special chars, you will need to escape them properly later on in the `docker-compose.yml` file for Vaultwarden. - If you are not sure about the subnet required for the user’s privileges, you can just use ‘`%`’ instead of specifying the IP range. ```sql mysql -u root -p CREATE DATABASE vaultwarden CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; CREATE USER 'vaultwarden'@'192.168.%' IDENTIFIED BY 'your_secure_password'; GRANT ALL PRIVILEGES ON vaultwarden.* TO 'vaultwarden'@'192.168.%'; FLUSH PRIVILEGES; ```
### Install dependencies + Vaultwarden itself - Take a snapshot of each web server before you start the process! - Install the following on EACH web node you have: ```bash # Update and install basic tools sudo apt update && sudo apt install -y ca-certificates curl # Add Docker's official GPG key sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null # Refresh apt & install Docker sudo apt update sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin # Enable Docker Systemd service (so containers start on boot) sudo systemctl enable --now docker ``` - Create a user, folder and a Docker Compose file: ```bash # Create the directory sudo mkdir -p /opt/vaultwarden # Set permissions so your current user can edit files there sudo chown $USER:$USER /opt/vaultwarden cd /opt/vaultwarden # Create the Compose file nano /opt/vaultwarden/docker-compose.yml ``` - The `compose.yml` file: - Note that in my setup, I have a virtual IP set up using OPNSense that I point the web application onto. If you do not have it set up yet, you can check out this part of the tutorial titled [**Virtual IP Set Up on OPNSense**](https://bachelor-tech.com/detailed-guides/deploy-mariadb-galera-cluster-on-proxmox/configure-haproxy-for-your-galera-cluster/#Virtual_IP_Set_Up_on_OPNSense) (you will then need a service such as HAProxy listening on that IP to receive and forward requests to your Galera cluster nodes). ```yaml services: vaultwarden: image: vaultwarden/server:latest container_name: vaultwarden restart: always ports: - "127.0.0.1:8000:80" environment: - DATABASE_URL=mysql://vaultwarden:yourpwd@1.2.3.4/vaultwarden - DOMAIN=https://vault.yourdomain.com - SIGNUPS_ALLOWED=false - INVITATIONS_ALLOWED=true - IP_HEADER=X-Real-IP - LOG_LEVEL=info - EXTENDED_LOGGING=true # --- SMTP Email Settings --- - SMTP_HOST=smtp.gmail.com # Replace with your provider's host - SMTP_FROM=vaultwarden@your-domain.tld - SMTP_FROM_NAME=Vaultwarden - SMTP_SECURITY=starttls # Options: starttls, force_tls, off - SMTP_PORT=587 # Usually 587 for starttls, 465 for force_tls - SMTP_USERNAME=your_email@gmail.com - SMTP_PASSWORD=your_app_password - SMTP_AUTH_MECHANISM="Plain" # This is safe if used with TLS logging: driver: "journald" options: tag: "{{.Name}}" volumes: # This creates a 'vw-data' sub-folder as a form of persistent storage - ./vw-data:/data ``` - Run it: ```bash cd /opt/vaultwarden # Sudo is needed to pull the image sudo docker compose up -d ``` - You can check its status after creation by running `sudo docker container ps -a` . > 💡 **Note** > > Once your **`docker-compose.yml`** file is launched and you make changes to it, it is best to then re-create the container by running **`sudo docker compose up -d --force-recreate`** ### Configure Nginx with Vaultwarden - Depending on how your nginx instance is already configured, your setup may look something like this: - Ensure the path + port number match! - SSL offloading is handled by HAProxy/OPNSense, so Nginx here listens on 8081. Ensure HAProxy passes the `X-Forwarded-Proto` header so Vaultwarden knows it's secure. ```bash sudo nano /etc/nginx/conf.d/vaultwarden.conf server { # We listen on 80 because HAProxy handles the SSL/HTTPS before it gets here. listen 8081; # Replace with your actual FQDN server_name vault.yourdomain.com; # Replace with OPNSense's actual IP to ensure fail2ban blocks the correct users: set_real_ip_from 192.168.0.0/16; real_ip_header X-Forwarded-For; real_ip_recursive on; # Allow large attachments (optional, but good for saving PDFs/Images in vault) client_max_body_size 128M; location / { proxy_pass http://127.0.0.1:8000; # WebSocket support (used by /notifications/hub) proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; # Pass headers so Vaultwarden knows the real IP of the user, not just "127.0.0.1" proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } ``` - To clarify the ports used (these remain internal and do not need firewall rules): - **Port 8081**: What nginx listens on to receive traffic from HAProxy - **Port 8000**: The Vaultwarden application (web interface, API, and WebSockets) > 💡 Note: In Vaultwarden versions prior to 1.29.0, WebSockets required a separate > port (3012). Modern versions serve WebSockets from the main application port, > simplifying the configuration. - Check nginx config and reload it: ```bash # Confirm there is no typo in the syntax before reloading it sudo nginx -t # Reload nginx for the changes to take an effect sudo systemctl reload nginx ``` - Remember to complete the above on all web server nodes, although you may wish to do the first one first and have HAProxy set up just with that one node until you got it working fully. --- ## 3. Configure OPNSense + HAProxy for Vaultwarden - For true High Availability (ideally on each site or at least on the main site), you should [**run your OPNSense in a master-backup configuration**](https://bachelor-tech.com/detailed-guides/opnsense-in-ha-with-carp-with-dual-wans/) running on separate hardware on the same site. - You would need to open a port for HTTPS communication and since your OPNSense HA relies on having the web interface running on port 443, you will need to open another port on HAProxy to receive web traffic on (I use port 4443 that the ISP-provided router changes from 443 to 4443). See here for [**more details on which ports to open for web traffic to work**](https://bachelor-tech.com/tutorials/iredmail-mail-server-as-proxmox-vm-with-opnsense/firewall-ports-to-open-on-your-firewall-opnsense/) on OPNSense, followed up by [**NAT rules**](https://bachelor-tech.com/tutorials/iredmail-mail-server-as-proxmox-vm-with-opnsense/fiirewall-set-up-nat-rules-opnsense/). - Create a DNS record for your `vault.yourdomain.tld` record pointing to the public IP of your OPNSense - you can [**follow these steps to set up dynamic DNS with Cloudflare**](https://bachelor-tech.com/tutorials/iredmail-mail-server-as-proxmox-vm-with-opnsense/dynamic-dns-for-our-mail-dns-record-cloudflare-with-opnsense/). - If you prefer a more secure set up, you do not actually need to expose a public DNS record, you can simply set one up on your network using tools such as Unbound DNS. - Before setting up HAProxy, you should also [**set up an SSL certificate using the ACME plugin**](https://bachelor-tech.com/tutorials/iredmail-mail-server-as-proxmox-vm-with-opnsense/get-ssl-certificate-on-opnsense-for-web-services-cloudflare/) in OPNSense. ### Add your ‘real’ hosts In case you do not have it set up yet, apart from defining your web hosts (under the ‘Real Servers’ tab) with the local port (I’m using 8081), we should check the health of each web host. - In OPNSense, go to Services → HAProxy → Settings. Then move to the other menu and click on Real Servers → Real Servers. **Add a new one for each of your web servers**: - Name: `web1_nginx` - Type: `static` - IP: your actual IP - Port: your chosen port configured on nginx, e.g. `8081` - No SSL config here! ### Set up a health monitor We should also ensure that there is a health check in place for the web servers. - Go to Rules & Checks → Health Monitors. Add a new monitor: - Name: `web_server_cluster_check` (or whatever you prefer) - Check type: `HTTP (default)` - SSL preferences: `use server settings` (default) - Check interval: `15-90s` (you may get some false positives if too aggressive) - Port to check: `8081` - HTTP method: `HEAD` - Request URI: `/` - HTTP version: `HTTP/1.1` - HTTP host: `yourdomain.tld` ### Configure your back-end pool You may already have a back-end pool configured for your existing web servers, yet we will need to create another pool with the same real hosts in it, since different parameters will need to be set up for Vaultwarden. > 💡 **Note** > > The challenge with the default session rate period is that **it** is typically about 10 seconds long as well as no pass-through option is defined, which would result in the client having to re-establish connection with the server (users would see having a ‘Connecting…’ spinning wheel appear regularly). While for example PHP-based applications receive a payload and close the connection after responding, for web sockets, we want the session to remain active. - Create a new back-end pool as follows (enable the advanced options): - Name: `backend_vaultwarden_nginx` - Mode: `HTTP (Layer 7)` - Balancing Algorithm: `Round Robin` - Random Draws: `2` (entirely up to you) - Proxy Protocol: none - Servers: add your ‘real’ servers here - … - Enable health checking: `Ticked` - Health monitor: `web_server_cluster_check` (previously defined in the health check section) - Proxy Protocol: `none` - Servers: add your web servers - … - X-Forwarded-For header: `Ticked` - Persistence Type: `Cookie-based persistence` - Cookie Handling: `Insert new cookie` - Cookie name: `SRVCOOKIE` (or another name as you desire) - Strip Quotes: `Ticked` (Default) - Expiration Time: `1h` (match your tunnel timeout) - … - Connection Timeout: `5s` (fail-over fast if the server is not responding) - Check Timeout: `5s` (fail fast if health check fails) - Server Timeout: `3600s` (allows the server to keep the pipe open) - Retries: `3` (default) - Option pass-through: `timeout tunnel 3600s` (tells HAProxy that it is a tunnel) - You can leave the rest as default unless you have other specifics in your setup. ### Configure a condition and a rule on HAProxy to listen to - While still under HAProxy’s settings, go to Rules & Checks → Conditions and add a new condition: - Name: `condition-vault-your-domain-tld` - Condition type: `Host matches` - Host string: `vault.yourdomain.tld` - And then still under Rules & Checks, go to Rules and add a new rule: - Name: vault-your-domain-tld_match (do not use spaces) - Test type: `IF` - Select conditions: `condition-your-domain-tld` (find your own) - Execute function: Use specified Backend Pool - Use backend pool: `backend_vaultwarden_nginx` (as defined previously) ### Set up your public (front-end) service Go to Virtual Services → Public Services. If you already have one defined for HTTPS, then simply add the rule at the end, test & apply the syntax and you are done. - If you have not set up HTTPS, then create a new one. The key here is to firstly have firewall rules set up to forward traffic to ‘This host’ on a port that HAProxy is listening to (must be different from the web interface that you use to access OPNSense!) and then in the ‘Public Service’ window, tick the ‘`SSL offloading`’ box to ensure that traffic is decrypted on your LAN. You would need to use the ACME certificate created earlier. > 💡 **Note** > > Ensure that you follow these steps on all of your sites to have the same config and can thus expect consistent behavior. --- ## 4. Troubleshoot Vaultwarden Docker/Web UI service Let’s try accessing the web UI. Try accessing the Vaultwarden interface on [`https://vault.yourdomain.com`](https://vault.yourdomain.com/) . ### Troubleshooting web UI reachability In case the web UI is not coming up, we need to determine where the traffic got stuck. Here are the layers: - `DNS` - e.g. CloudFlare - is your DNS record set up? Does it point to the correct IP address? - `OPNSense` - do you have a firewall + NAT rule set up for forwarding traffic to a port that HAPRoxy is listening on? - `HAProxy` - is your public (front-end) service on and listening on the desired port? Have you attached your SSL certificate for decrypting traffic? Is your back-end pool forwarding traffic to the right host? Is your ‘real’ host configure on the right port? - `Nginx` on the VM - is nginx on and listening on the right port? Is the syntax of your virtual host for forwarding ok? - `Docker` - is the docker instance up? What do the logs say? - `Galera DB Cluster` - is it reachable for the web servers? Is the virtual IP operational? Is the DB up, esp. if you are reaching it on a virtual IP via OPNSense / pfSense? ### Example issue no.1 - Web server not accepted by the DB server The example below shows that the Docker instance is running but spent the last two hours in a crash loop due to incorrect DB details: - You may notice the IP 192.168.8.1 that points to the OPNSense host from which it is forwarded. This is because the Galera cluster is reachable on a virtual IP (192.168.8.70) provided by OPNSense. Such behavior is normal and expected. - The main issue that I simulated here was that the user was defined for the wrong subnet - after dropping the user and re-creating it with the correct GRANT privileges fixed the issue. ### Example issue no.2 - No Docker container running Let’s say you set up just one web server VM and were able to reach the web UI and then later, it stopped loading. You may wish you double check that the docker container is still up. ```bash sudo docker container ps -a ``` - To fix it, go to the folder where your container is located and start it. Ensure that you set up Docker to start on boot. ```bash cd /opt/vaultwarden sudo docker compose up -d sudo systemctl enable docker ``` ### Example issue no.3 - "401 Unauthorized" loops If you log in successfully but are immediately logged out when refreshing or performing an action, your nodes likely have different signing keys. - **The Cause:** Vaultwarden generates RSA key pair files in `/data` on first startup. If `web1` generated its own keys and `web2` generated different ones, a login token signed by `web1` will be rejected by `web2`. - **The Fix:** Ensure Syncthing is successfully syncing the `rsa_key.pem`, `rsa_key.pub.pem`, and `rsa_key.der` files. You may need to manually copy the keys from the "primary" node to the others once to establish a baseline, then let Syncthing handle future updates. ### First login - Your first step in the Web UI would be to create your account. - Since we set `SIGNUPS_ALLOWED=false` in the config, you will need to temporarily change that to `true` in `docker-compose.yml`, run `docker compose up -d` to apply, create your accounts, and then flip it back to `false` and restart again. This is intentional as a matter of exercising how you can easily tweak settings with Docker Compose, esp. when applying updates in the future. --- ## 5. Set up Syncthing for Vaultwarden data sync To ensure that the uploaded data stay consistent across multiple web servers, we will employ Syncthing, which I already use on my web servers. If you would like some help with deploying it, check out my [**previous tutorial on How to Configure HA for Web Servers**](https://bachelor-tech.com/tutorials/how-to-create-high-availability-for-a-web-server-using-syncthing-and-haproxy/). - The permissions depend on which user runs the Syncthing service. In my case, this is `www-data`. We will need to ensure that this user has access to Docker for monitoring purposes: ```yaml # Stop the container briefly to ensure the folder is free: cd /opt/vaultwarden sudo docker compose down # Grant the 'www-data' user full Read/Write access to the docker volume # using ACLs (Access Control Lists). This persists even if Docker recreates files. sudo setfacl -R -m u:www-data:rwx /opt/vaultwarden/vw-data sudo setfacl -d -m u:www-data:rwx /opt/vaultwarden/vw-data # Start the container again (in a detached mode to free up the shell): sudo docker compose up -d ``` - Assuming you have Syncthing already deployed, add a new folder on web1: - **General** tab: - Label: Vaultwarden (web1) - Path: `/opt/vaultwarden/vw-data` - **Sharing** tab → if you already have your other Syncthing web servers defined, then tick the boxes. - **File versioning** tab → It is recommended to set up ‘Trash can versioning’ for 30 days+. - Skip to the **Advanced** tab → Check **[x] Ignore Permissions** (Docker manages the ownership, Syncthing just moves the data bits). - Go back to the **Ignore Patterns** tab: (tick the box and once you complete the other tabs and go forward, copy paste this list): ```bash icon_cache/ tmp/ temp/ *.sqlite3 *.sqlite3-wal *.sqlite3-shm *.log ``` ### Troubleshooting Syncthing folder addition - In case you add the sync job and immediately get a permission error like the one below, it means your permissions on the `/opt/vaultwarden` folder are not correct: - While in my case, the user is www-data, in your case, it is likely different. How can you find out? Let’s take a look: ```bash # List all running processes, filter 'syncthing' and filter out the command itself: sudo ps aux | grep syncthing | grep -v grep ``` - Then adjust the ‘`sudo setfacl -R`’ and `‘sudo setfacl -d`’ commands above accordingly. - For additional troubleshooting scenarios, you can check my [**previous guide for Syncthing**](https://bachelor-tech.com/tutorials/how-to-create-high-availability-for-a-web-server-using-syncthing-and-haproxy/) that includes data loss situations and recovery options. ### Add Syncthing on other web nodes - SSH into your other web server nodes and set up the file permissions accordingly as we have done at the beginning of this Step. - Open Syncthing on each of your other web nodes and accept the invitation: - Set up the name, path, trash can, ignore permissions in the Advanced tab and then set up the file/folder patterns to be ignored, as we have done previously. --- ## 6. Set up Monitoring for Vaultwarden’s Docker Container + Website using UptimeKuma There are three types of monitors that would be good to set up. If you only have one site or do not have your ‘witness’ site set up yet, then you can have all these checks running of just one UptimeKuma instance. ### Monitor A. Site 3 - Website Uptime On Site 3 (VPS - witness site), you can monitor the service as a whole (e.g. does the website load regardless of which site it is being loaded from?). - **Monitor Type**: `HTTP(s)` - **Friendly name**: `Vaultwarden or similar` - **URL**: Your actual URL - **Heartbeat**: 30-600 seconds (1 to 10 minutes) - **Retries**: 1-3 - **Heartbeat Retry:** a small amount, such as 30-60 seconds ### Monitor B. Site 1+2 Docker Container Monitor For UptimeKuma to see the container, we will need to deploy a **Socket Proxy container. ** The idea is to use the proxy as a ‘gatekeeper’ that provides read-only access to UptimeKuma on a specific port. We will need to apply this on **each web server node**. - Create a new folder and a compose file: ```bash # Create a directory for the proxy sudo mkdir -p /opt/socket-proxy-container # Go into that directory cd /opt/socket-proxy-container # Create the compose file sudo nano docker-compose.yaml ``` - Enter the following: ```yaml services: docker-socket-proxy: image: tecnativa/docker-socket-proxy container_name: docker-socket-proxy restart: unless-stopped # High privilege is required to access the raw docker socket privileged: true ports: - "2375:2375" volumes: # We give it access to the host's docker socket (read-only) - /var/run/docker.sock:/var/run/docker.sock:ro environment: # ENABLE specific read-only permissions - CONTAINERS=1 # Allows listing and checking container status - INFO=1 # Allows checking general docker info (optional) # BLOCK write permissions (security) - POST=0 # Blocks any commands that change state (restart/kill/create) ``` > 💡 Security Note: The socket proxy exposes read-only Docker API access on port 2375. > In a homelab environment on a trusted LAN, this is generally acceptable. For stricter > security, bind to a specific IP (`192.168.8.9:2375:2375`) or implement firewall rules > to allow only your UptimeKuma host on port `2375`. The most secure method would be to bind it to `localhost` only and use SSH tunneling, instead. - Then start the container: ```bash cd /opt/socket-proxy-container sudo docker compose up -d # Check that it is running together with the Vaultwarden container: sudo docker ps -a ``` - On Site1/Site2 UptimeKuma, add a new monitor: - **Monitor Type**: `Docker Container` - **Friendly name**: `Web1 - Docker Monitor - Vaultwarden` - **Container name**: `vaultwarden` (as per how it’s shown when you run `docker container ps -a` on your web server) - **Docker host** - add a new one: - Friendly name: `Web1 Docker Host` - Connection type: `TCP / HTTP` - Docker Daemon: `tcp://192.168.8.9:2375` (use your local IP with port 2375) - The overall monitor for **`web1`** may look like this - remember to repeat this for each web server: ### Monitor C. Site 1+2 Syncthing Monitor The other thing that could stop working over time is Syncthing itself. We can leverage Syncthing’s API for that and push information regularly from each web server into UptimeKuma as a passive ‘push’ monitor. - Since I have described this method already in my recent guide, please follow the steps from the [**Syncthing Web HA tutorial**](https://bachelor-tech.com/tutorials/how-to-create-high-availability-for-a-web-server-using-syncthing-and-haproxy/6-monitor-syncthing-jobs-with-uptimekuma/). - You will get notified whenever ANY of the sync jobs are down or even paused, as per your config settings. --- ## 7. Harden Vaultwarden with Fail2ban On the VM that runs docker with our Vaultwarden container, we should install fail2ban and secure the docker container. In case you do not have it set up already, you can also secure SSH and your other sites. ### Install fail2ban ```bash # 1. Install Fail2ban sudo apt update sudo apt install fail2ban -y # 2. Create a local configuration file (never edit jail.conf directly) sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local ``` ### Protect SSH on a custom port - In this case, SSH port is set to a custom port 2222 under `/etc/ssh/sshd_config` . ```bash sudo nano /etc/fail2ban/jail.local # Un-commment these: ignoreself = true # Add your own IP range (oneself + LAN + docker subnet) ignoreip = 127.0.0.1/8 ::1 192.168.0.0/16 172.20.0.0/16 # Comment out # ignorecommand = # Find an existing section called [sshd] and modify it as follows: [sshd] enabled = true port = 2222 mode = aggressive # The more modern method instead of reading text files backend = systemd maxretry = 3 bantime = 1h ``` - If you have not started the fail2ban client yet, run `sudo fail2ban-client start` . Otherwise run `sudo fail2ban-client reload` for the changes above to kick in. ### Protect Vaultwarden’s docker container - In your docker-compose.yml file, ensure that you have the following parameters set: ```bash # Ensure these three are somewhere in your 'environment': environment: - LOG_FILE=/data/vaultwarden.log - LOG_LEVEL=info - EXTENDED_LOGGING=true ``` - After changing the values below, run `docker compose up -d` to apply changes. - Let’s create a config that instructs `fail2ban` on what a failed login to Vaultwarden looks like: ```bash sudo nano /etc/fail2ban/filter.d/vaultwarden.conf [Definition] # Matches "Admin login attempt failed" and "Invalid password for user" failregex = .*Username or password is incorrect.*IP: