# Automating Vaultwarden Docker Updates Across Multiple Nodes [TOC] This is a follow up to my tutorial on deploying Vaultwarden in a multi site HA setup (see the other .md file in this folder). Once you have Vaultwarden running behind Docker Compose on two or three nodes, the next problem you run into is maintenance: someone has to notice a new release exists, pull it, and restart the container on every node. This is important because certain new releases break certain functionality, mostly due to having to port them back to Bitwarden-driven updates. This guide covers a small bash script that checks GitHub daily for a new Vaultwarden release, applies it safely with a health check and automatic rollback, and pings you on Discord (or wherever you want) when it does. It runs independently on each node, no central controller needed. ## Prerequisites - Vaultwarden already deployed via Docker Compose on each node, with a `docker-compose.yml` at a known path (this guide assumes `/opt/vaultwarden`). - SSH access to each node with a user that can run `sudo docker` and `sudo cron` related commands without needing a manual password every time, since this needs to run unattended overnight. If your web nodes are managed through something like Ansible, they likely already have this set up. - `curl` and `jq` installed on each node (`sudo apt install -y curl jq` on Debian). - If you want notifications: a Discord webhook, an SMTP relay, or whatever else you want to shell out to. ## Step 1: Pin an explicit version instead of `:latest` If your compose file currently has this: ```yaml services: vaultwarden: image: vaultwarden/server:latest ``` change it to an explicit version: ```yaml services: vaultwarden: image: vaultwarden/server:1.37.2 ``` This matters because the auto update script needs a baseline to compare against. If the image is pinned to `latest`, there is no version string in the compose file to read, so the script has nothing to diff the newest GitHub release against. Pinning also means a `docker compose pull` never silently changes your running version outside of the update window you control. Do the first pin and update manually, the normal way: ```bash cd /opt/vaultwarden sudo docker compose pull sudo docker compose up -d sudo docker ps --filter name=vaultwarden ``` Confirm the container reports healthy and the app responds before moving on: ```bash curl -s http://127.0.0.1:8000/alive ``` ## Step 2: Decide your update policy This is the one decision worth thinking about before you automate anything. Vaultwarden runs a database schema migration on container startup. If you run a single node, that is a non issue. If you run multiple nodes against one shared database (Galera, a managed MySQL cluster, whatever), a **minor or major** version bump (1.37.x to 1.38.0) migrated by node A while node B and C are still running the old binary is exactly the kind of cross version conflict that can cause weird errors until every node catches up. A **patch** bump (1.37.2 to 1.37.3) essentially never touches the schema, so it is low risk to apply automatically everywhere. Two reasonable policies: - **Patch only, auto apply. Notify on anything bigger.** Safer for shared database setups. You get pinged when a minor or major release exists and apply it yourself after a snapshot. - **Apply everything automatically.** Simpler, but accept that for a short window (until every node's cron has run) some nodes might be on a newer schema than others. In practice this has been fine for me, but it is a real tradeoff, not a hypothetical one, so make the choice on purpose rather than by default. The script below supports both, controlled by a single variable. ## Step 3: The auto update script Save this as `/usr/local/sbin/vaultwarden-autoupdate.sh` on each node, owned by root, mode `0755`. What it does, in order: 1. Reads the currently pinned version out of `docker-compose.yml`. 2. Asks the GitHub releases API what the newest tag is, with retries, and refuses to act on anything that is not a plain `X.Y.Z` version string (the tag is untrusted input coming off the network, and it ends up in a `docker pull` and a config file, so it gets validated before anything else happens). 3. Compares versions. If nothing is newer, it exits quietly. 4. If `UPDATE_POLICY` is set to `patch` and the new release is a minor or major bump, it sends a "held back" notification and stops, without touching anything. 5. Otherwise, it pulls the new image first, before changing any config, so a bad tag or a registry outage is caught before your compose file is touched. 6. Backs up the compose file, rewrites the image tag, validates the compose file still parses, and recreates the container. 7. Waits for the container to report healthy (or, if the image has no health check, probes the app directly). 8. If it becomes healthy, logs success and sends a notification naming the version and the hostname. 9. If it does not become healthy within the timeout, it automatically restores the previous compose file, brings the old version back up, confirms that comes back healthy too, and sends a failure notification. If even the rollback fails, it sends a distinct "this node is down, go look at it now" alert rather than pretending everything is fine. A lock file stops two runs from overlapping if a previous run somehow hangs. ```bash #!/usr/bin/env bash # # vaultwarden-autoupdate.sh # # Checks GitHub once a day for the newest Vaultwarden release. If it is newer # than the tag currently pinned in docker-compose.yml, it pulls the image, # rewrites the pinned tag, recreates the container and verifies it comes back # healthy. If it does not come back healthy, it rolls back to the previous tag. # # Runs independently on each web node; no coordination between nodes. # # Install: /usr/local/sbin/vaultwarden-autoupdate.sh (root:root, 0755) # Log: /var/log/vaultwarden-autoupdate.log + journald tag "vw-autoupdate" # # Manual use: # vaultwarden-autoupdate.sh # normal run # vaultwarden-autoupdate.sh --dry-run # report only, change nothing # vaultwarden-autoupdate.sh --force # re-apply even if already current set -uo pipefail # ------------------------------- configuration ------------------------------- COMPOSE_DIR="/opt/vaultwarden" COMPOSE_FILE="${COMPOSE_DIR}/docker-compose.yml" CONTAINER="vaultwarden" # container_name inside docker-compose.yml IMAGE_REPO="vaultwarden/server" GITHUB_REPO="dani-garcia/vaultwarden" HEALTH_URL="http://127.0.0.1:8000/alive" # UPDATE_POLICY: # all - apply any newer release, including minor/major bumps (1.37.x -> 1.38.0) # patch - apply only patch bumps (1.37.2 -> 1.37.3); newer minor/major releases # are logged as "held back" and left for a manual, snapshotted update. # # Note: if multiple nodes share one database and Vaultwarden applies schema # migrations on startup, a minor/major bump applied by one node migrates the # shared schema while the other nodes are still running the old binary. Set # this to "patch" if you would rather do those bumps by hand. UPDATE_POLICY="all" # Seconds to wait for the container to report healthy before declaring failure. HEALTH_TIMEOUT=180 # Optional notification hook, invoked as: NOTIFY_CMD # Leave empty to disable. See the Discord example script further down. NOTIFY_CMD="/usr/local/sbin/vw-notify-discord.sh" LOG_FILE="/var/log/vaultwarden-autoupdate.log" LOCK_FILE="/var/lock/vaultwarden-autoupdate.lock" BACKUP_DIR="${COMPOSE_DIR}/.autoupdate-backups" # ------------------------------- plumbing ------------------------------------ DRY_RUN=0 FORCE=0 for arg in "$@"; do case "$arg" in --dry-run) DRY_RUN=1 ;; --force) FORCE=1 ;; -h|--help) sed -n '2,19p' "$0"; exit 0 ;; *) echo "unknown argument: $arg" >&2; exit 2 ;; esac done log() { local level="$1"; shift local msg="$*" printf '%s [%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$level" "$msg" >>"$LOG_FILE" logger -t vw-autoupdate -- "$msg" 2>/dev/null || true if [ -t 1 ]; then printf '[%s] %s\n' "$level" "$msg"; fi return 0 } notify() { if [ -z "$NOTIFY_CMD" ]; then return 0; fi "$NOTIFY_CMD" "$1" "$2" >/dev/null 2>&1 || log WARNING "notification hook failed" return 0 } die() { log ERR "$*" notify "Vaultwarden auto-update FAILED on $(hostname -s)" "$*" exit 1 } # Compare dotted versions. ver_gt A B -> true if A is strictly newer than B. ver_gt() { if [ "$1" = "$2" ]; then return 1; fi [ "$(printf '%s\n%s\n' "$1" "$2" | sort -V | tail -n1)" = "$1" ] } compose() { docker compose --project-directory "$COMPOSE_DIR" -f "$COMPOSE_FILE" "$@" } wait_healthy() { local deadline=$((SECONDS + HEALTH_TIMEOUT)) state health while [ "$SECONDS" -lt "$deadline" ]; do state="$(docker inspect -f '{{.State.Status}}' "$CONTAINER" 2>/dev/null)" || state="" health="$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$CONTAINER" 2>/dev/null)" || health="" if [ "$state" = "running" ]; then case "$health" in healthy) return 0 ;; none) # No HEALTHCHECK in the image, probe the application directly. if curl -fsS -o /dev/null --max-time 10 "$HEALTH_URL"; then return 0; fi ;; unhealthy) return 1 ;; esac elif [ "$state" = "exited" ] || [ "$state" = "dead" ]; then return 1 fi sleep 5 done return 1 } # ------------------------------- preflight ----------------------------------- umask 077 touch "$LOG_FILE" 2>/dev/null || { echo "cannot write $LOG_FILE" >&2; exit 1; } chmod 640 "$LOG_FILE" 2>/dev/null || true # Serialise against a previous run that is still going. exec 9>"$LOCK_FILE" || die "cannot open lock file $LOCK_FILE" flock -n 9 || { log INFO "another run is in progress; exiting"; exit 0; } if [ "$(id -u)" -ne 0 ]; then die "must run as root"; fi if [ ! -f "$COMPOSE_FILE" ]; then die "compose file not found: $COMPOSE_FILE"; fi command -v docker >/dev/null || die "docker not found" command -v curl >/dev/null || die "curl not found" docker compose version >/dev/null 2>&1 || die "docker compose plugin not available" mkdir -p "$BACKUP_DIR" && chmod 700 "$BACKUP_DIR" # ------------------------- current and latest version ------------------------ CURRENT="$(grep -oP "^\s*image:\s*${IMAGE_REPO}:\K[^\s#]+" "$COMPOSE_FILE" | head -n1)" if [ -z "$CURRENT" ]; then die "could not read the pinned image tag from $COMPOSE_FILE"; fi if [ "$CURRENT" = "latest" ]; then die "compose file pins the floating 'latest' tag, not a version. Pin an explicit version first (e.g. image: ${IMAGE_REPO}:1.37.2) so this script has a baseline to compare against." fi log INFO "current pinned version: ${CURRENT} (policy=${UPDATE_POLICY})" API="https://api.github.com/repos/${GITHUB_REPO}/releases/latest" RESPONSE="" for attempt in 1 2 3; do RESPONSE="$(curl -fsSL --max-time 30 -H 'Accept: application/vnd.github+json' -H 'User-Agent: vaultwarden-autoupdate' "$API" 2>/dev/null)" && break log WARNING "GitHub API attempt ${attempt}/3 failed; retrying" sleep $((attempt * 10)) done if [ -z "$RESPONSE" ]; then die "could not reach the GitHub releases API after 3 attempts"; fi LATEST="$(printf '%s' "$RESPONSE" | grep -oP '"tag_name"\s*:\s*"\K[^"]+' | head -n1)" LATEST="${LATEST#v}" # The tag is untrusted input from the network and ends up in a config file and a # docker pull, so accept nothing but a plain X.Y.Z version. if ! [[ "$LATEST" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then die "GitHub returned a tag that is not a plain X.Y.Z version: '${LATEST}' - refusing to act on it" fi log INFO "latest release on GitHub: ${LATEST}" # ------------------------------ decide --------------------------------------- if [ "$FORCE" -eq 1 ]; then log INFO "--force given; applying ${LATEST} regardless of current version" elif ! ver_gt "$LATEST" "$CURRENT"; then log INFO "already up to date (${CURRENT}); nothing to do" exit 0 elif [ "$UPDATE_POLICY" = "patch" ] && [ "${LATEST%.*}" != "${CURRENT%.*}" ]; then log NOTICE "held back: ${CURRENT} -> ${LATEST} is a minor/major bump and UPDATE_POLICY=patch. Snapshot the VM and update manually." notify "Vaultwarden ${LATEST} available on $(hostname -s) (held back)" "Node $(hostname -s) is running ${CURRENT}. Version ${LATEST} is available but was not applied automatically because it is a minor/major bump and UPDATE_POLICY=patch. If you run multiple nodes against a shared database, apply this one deliberately: snapshot the VM, then run vaultwarden-autoupdate.sh --force" exit 0 fi log NOTICE "update available: ${CURRENT} -> ${LATEST}" if [ "$DRY_RUN" -eq 1 ]; then log INFO "--dry-run given; stopping here without changing anything" exit 0 fi # ------------------------------ apply ---------------------------------------- # Pull first. If the tag does not exist or the registry is unreachable we find # out before the compose file has been touched. log INFO "pulling ${IMAGE_REPO}:${LATEST}" if ! docker pull --quiet "${IMAGE_REPO}:${LATEST}" >/dev/null 2>&1; then die "failed to pull ${IMAGE_REPO}:${LATEST}; compose file left unchanged" fi STAMP="$(date +%Y%m%d-%H%M%S)" BACKUP="${BACKUP_DIR}/docker-compose.yml.${CURRENT}.${STAMP}" cp -p "$COMPOSE_FILE" "$BACKUP" || die "could not back up the compose file" log INFO "compose file backed up to ${BACKUP}" restore_and_exit() { local reason="$1" log ERR "$reason" log NOTICE "rolling back to ${CURRENT}" if cp -p "$BACKUP" "$COMPOSE_FILE" && compose up -d >>"$LOG_FILE" 2>&1; then if wait_healthy; then log NOTICE "rollback to ${CURRENT} succeeded; container is healthy" notify "Vaultwarden auto-update FAILED on $(hostname -s), rolled back" "Update ${CURRENT} -> ${LATEST} failed and was rolled back. The container is back on ${CURRENT} and healthy. Reason: ${reason} Log: ${LOG_FILE}" exit 1 fi fi log EMERG "ROLLBACK FAILED - Vaultwarden is DOWN on $(hostname -s), manual intervention needed" notify "Vaultwarden DOWN on $(hostname -s) - rollback failed" "Update ${CURRENT} -> ${LATEST} failed AND the rollback failed. Vaultwarden is not healthy on this node. Manual intervention required. Reason: ${reason} Compose backup: ${BACKUP} Log: ${LOG_FILE}" exit 1 } # Rewrite only the image line for this repo, matched at the start of the line. if ! sed -i -E "s|^([[:space:]]*image:[[:space:]]*)${IMAGE_REPO}:[^[:space:]#]+|\1${IMAGE_REPO}:${LATEST}|" "$COMPOSE_FILE"; then cp -p "$BACKUP" "$COMPOSE_FILE" die "failed to rewrite the image tag; compose file restored" fi NEW_PINNED="$(grep -oP "^\s*image:\s*${IMAGE_REPO}:\K[^\s#]+" "$COMPOSE_FILE" | head -n1)" if [ "$NEW_PINNED" != "$LATEST" ]; then restore_and_exit "compose file still pins '${NEW_PINNED}' after the rewrite" fi # Catch YAML damage before it reaches the running container. compose config --quiet >>"$LOG_FILE" 2>&1 || restore_and_exit "docker compose config rejected the rewritten file" log INFO "recreating the container on ${LATEST}" compose up -d >>"$LOG_FILE" 2>&1 || restore_and_exit "docker compose up -d failed" if ! wait_healthy; then docker logs --tail 50 "$CONTAINER" >>"$LOG_FILE" 2>&1 || true restore_and_exit "container did not become healthy within ${HEALTH_TIMEOUT}s on ${LATEST}" fi RUNNING="$(docker inspect -f '{{index .Config.Labels "org.opencontainers.image.version"}}' "$CONTAINER" 2>/dev/null)" log NOTICE "update complete: ${CURRENT} -> ${LATEST} (container reports '${RUNNING:-unknown}'), healthy" notify "Vaultwarden updated to ${LATEST} on $(hostname -s)" "Node $(hostname -s) was updated from ${CURRENT} to ${LATEST} and is healthy. Compose backup: ${BACKUP} Log: ${LOG_FILE}" # Keep the last 10 compose backups. ls -1t "${BACKUP_DIR}"/docker-compose.yml.* 2>/dev/null | tail -n +11 | xargs -r rm -f exit 0 ``` ## Step 4: Notifications The script calls whatever you set `NOTIFY_CMD` to, passing it a subject and a body as two arguments. That keeps the notification channel pluggable, swap the script below for one that hits Slack, sends an email through `mailx`, or whatever you already use. Here is the Discord version, since it took two minutes to set up and just works. In Discord: Server Settings > Integrations > Webhooks > New Webhook, pick the channel, copy the URL. Save this as `/usr/local/sbin/vw-notify-discord.sh`, owned by root, mode `0700` (the webhook URL is a bearer credential, anyone holding it can post to your channel, so keep this file locked down). ```bash #!/usr/bin/env bash # # vw-notify-discord.sh # # Posts a message to a Discord channel via an incoming webhook. Used as the # NOTIFY_CMD hook for vaultwarden-autoupdate.sh. # # The webhook URL is a bearer credential (anyone holding it can post to the # channel) so this file is kept root-only, 0700. set -euo pipefail WEBHOOK_URL="https://discord.com/api/webhooks/REPLACE_ME/REPLACE_ME" SUBJECT="${1:-}" BODY="${2:-}" # Discord message content is capped at 2000 characters. MSG="**${SUBJECT}**"$'\n'"${BODY}" MSG="${MSG:0:1900}" PAYLOAD="$(jq -n --arg content "$MSG" '{content: $content}')" curl -fsS --max-time 10 \ -H 'Content-Type: application/json' \ -d "$PAYLOAD" \ "$WEBHOOK_URL" >/dev/null ``` Test it on its own before wiring it into anything: ```bash sudo /usr/local/sbin/vw-notify-discord.sh "test" "hello from $(hostname -s)" ``` ## Step 5: Deploy the scripts On each node: ```bash sudo install -o root -g root -m 0755 vaultwarden-autoupdate.sh /usr/local/sbin/vaultwarden-autoupdate.sh sudo install -o root -g root -m 0700 vw-notify-discord.sh /usr/local/sbin/vw-notify-discord.sh ``` ## Step 6: Schedule it Do not use `crontab -e` for this, it is easy to lose track of user crontabs across several nodes and forget they exist. Use a drop in file under `/etc/cron.d/` instead, it is one file you can `cat` to see exactly what is scheduled, and it is easy to copy across nodes with the same tooling you used to deploy the script. If you run more than one node against a shared database, stagger the times rather than running them all at once. That way if a run does need to roll back, or if a minor version bump does cause a brief cross version window, you are not doing it on every node simultaneously, and you have some separation to notice a problem on the first node before the second one runs. Create `/etc/cron.d/vaultwarden-autoupdate` on each node: ``` # Vaultwarden daily auto-update check 20 3 * * * root /usr/local/sbin/vaultwarden-autoupdate.sh >/dev/null 2>&1 ``` Pick a different minute/hour on each node, twenty minutes apart is plenty. My own three nodes run at 03:20, 03:40 and 04:00 respectively, all in the early morning low traffic window. Restart cron after adding the file so it picks it up: ```bash sudo systemctl restart cron ``` ## Step 7: Test before you trust it Run it by hand first, in dry run mode, to confirm it can talk to GitHub and read your compose file correctly: ```bash sudo /usr/local/sbin/vaultwarden-autoupdate.sh --dry-run ``` If you want to prove the whole update and rollback path actually works rather than assuming the code is correct, temporarily pin the compose file back a version, run the script for real, and watch it detect the gap and bring itself back up to date: ```bash sudo sed -i 's/image: vaultwarden\/server:.*/image: vaultwarden\/server:1.37.0/' /opt/vaultwarden/docker-compose.yml cd /opt/vaultwarden && sudo docker compose up -d sudo /usr/local/sbin/vaultwarden-autoupdate.sh ``` Check the log and confirm the container is healthy on the new version afterward: ```bash sudo tail -n 20 /var/log/vaultwarden-autoupdate.log sudo docker ps --filter name=vaultwarden ``` ## Gotchas - **Shared database, minor or major bumps.** Covered above, worth repeating: decide your `UPDATE_POLICY` on purpose rather than leaving it at whatever the default happens to be. - **A load balancer or firewall sitting between your app and your database needs its idle timeout set to match, not left at whatever the default is.** I found this one the hard way. One of my sites had its HAProxy backend pool for the database VIP left at the platform's default 30 second timeout, instead of matching the database's own 1800 second `wait_timeout` the way my other site's pool did. Every pooled connection was getting silently killed roughly once a minute, the app would reconnect and recover so it looked fine most of the time, but any request unlucky enough to land in that gap would fail. If you are running a similar setup, check that every backend pool fronting a long lived connection (a database, anything using persistent pooled connections) has an explicit, generous timeout rather than relying on whatever the default happens to be. A default that is fine for a quick HTTP request is often much too short for a pooled database connection. - **A flapping backend connection can corrupt more than just the request that fails.** In my case, a client syncing against a node with the timeout problem above ended up with a malformed record cached locally, which then crashed the app on every subsequent launch until the local cache was cleared. If a client app starts behaving strangely (crashing on render, refusing to log out, stuck spinners) right after a period of backend instability, clearing its local cache/data folder is worth trying before assuming the account itself is broken. - **Pull before you touch the compose file, not after.** If the pull fails (bad tag, registry down, whatever), you want to find that out before anything on disk has changed, so a failed run just does nothing rather than leaving you half migrated. - **Validate the version string coming back from GitHub before you use it anywhere.** It is network input. The script only accepts a plain `X.Y.Z` pattern before it goes anywhere near a `docker pull` or a config file. - **Use a lock file.** If a run somehow hangs (network stall, whatever), you do not want the next day's cron job to start a second one on top of it. ## Customizing this further - Swap `NOTIFY_CMD` for an email, Slack, or ntfy.sh script, the calling convention (`subject`, `body` as two arguments) stays the same regardless of the backend. - If you want a wider notice window before a minor/major update lands, set `UPDATE_POLICY=patch` and check your notification channel every so often, or schedule a second cron entry that just does the check. - If you are not on Debian/Ubuntu, replace the cron drop in with a systemd timer, the script itself does not care how it gets invoked.