# Part 4 - Create an n8n workflow for patching with Patchmon and Semaphore UI
[TOC]
In the previous parts of this series, we did the following:
- **In Part 1**, we set up Semaphore with useful playbooks for snapshotting hosts on Proxmox before patching and playbooks that patch hosts differently based on their groups (such as web and DB servers).
- **In Part 2**, we set up Patchmon, explained why it is not yet ready for patching despite its recently added capabilities and connected it to our fleet of hosts, whether on Proxmox or other. We pushed a Semaphore job that deployed the Patchmon agent on all standalone hosts and VMs.
- **In Part 3**, we stood up Uptime Kuma and tagged every monitor with the hostname it belongs to, so we can put a host's monitors into a maintenance window for the duration of a patch run, no false "down" alert from a reboot, or from the brief pause a Proxmox snapshot causes.
Now we are getting into the interesting part by connecting them together with n8n! While some may argue that βn8n is deadβ due to recently identified security vulnerabilities back in early 2026 and before, I would argue that if we can work around the resulting limitations (such as jobs being launched in isolated containers), we actually have a more robust, time-proven workflow engine with an okay UI. So letβs dive in!
## Step 1 - Deploy n8n
In case you do not have n8n installed already, here is a little guide on how to deploy it in a docker environment together with a containerized PostgreSQL instance to keep our data persistent and more production-ready than when using a simple SQLite (you can otherwise skip to Step 2).
- Create a folder on the VM for n8n and set up the docker-compose file:
```yaml
# Create directory and a docker-compose file
sudo mkdir -p /opt/n8n
cd /opt/n8n
sudo nano docker-compose.yml
```
```yaml
services:
postgres:
image: postgres:16.6-alpine
container_name: n8n-postgres
restart: unless-stopped
environment:
- POSTGRES_DB=n8n
- POSTGRES_USER=n8n
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
volumes:
- n8n_postgres_data:/var/lib/postgresql/data
networks:
- n8n-network
healthcheck:
test: ["CMD-SHELL", "pg_isready -U n8n -d n8n"]
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
# No ports published to the host on purpose - only n8n on this network needs it.
mem_limit: 512m
cpus: 1.0
n8n:
image: docker.n8n.io/n8nio/n8n:2.30.3
container_name: n8n
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
ports:
- "5678:5678"
environment:
# --- General ---
- GENERIC_TIMEZONE=Europe/Prague # Set up your timezone
- TZ=Europe/Prague # Set up your timezone
- NODE_ENV=production
# --- Database (Postgres, standalone, colocated with this container) ---
- DB_TYPE=postgresdb
- DB_POSTGRESDB_HOST=postgres
- DB_POSTGRESDB_PORT=5432
- DB_POSTGRESDB_DATABASE=n8n
- DB_POSTGRESDB_USER=n8n
- DB_POSTGRESDB_PASSWORD=${POSTGRES_PASSWORD}
# --- Credential encryption ---
# Set explicitly and back this up OUTSIDE the n8n_data volume.
# If this key is lost, every stored credential becomes unrecoverable.
- N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
# --- Networking ---
# NOTE: We will set up TLS later
- N8N_HOST=n8n.bakalar.priv
- N8N_PORT=5678
- N8N_PROTOCOL=http
- WEBHOOK_URL=http://1.2.3.4:5678/
- N8N_SECURE_COOKIE=false
# --- Code node permissions ---
# Loosened from defaults - our workflow does not require fs
- NODE_FUNCTION_ALLOW_BUILTIN=fs,path
- N8N_COMMUNITY_PACKAGES_ALLOW_TOOL_USAGE=true
- N8N_RESTRICT_ENVIRONMENT_VARIABLES_ACCESS=true
# --- Task runners ---
- N8N_RUNNERS_ENABLED=true
- N8N_RUNNERS_MAX_OLD_SPACE_SIZE=1024
- N8N_RUNNERS_TASK_TIMEOUT=600
volumes:
- n8n_data:/home/node/.n8n
- ./local-files:/files
networks:
- n8n-network
healthcheck:
test: ["CMD-SHELL", "wget -q --spider http://localhost:5678/healthz || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
mem_limit: 1g
cpus: 1.5
volumes:
n8n_data:
name: n8n_data
n8n_postgres_data:
name: n8n_postgres_data
networks:
n8n-network:
name: n8n-network
```
- Stand up a .env file - here is an example:
```bash
nano /opt/n8n/.env
# Copy this to .env (same directory as docker-compose.yml) and fill in real
# values. Do NOT commit .env to git - add it to .gitignore.
# Postgres password for the n8n database user.
POSTGRES_PASSWORD=change-me-to-a-strong-random-value
# n8n credential encryption key. Generate one with:
# openssl rand -base64 32
# Back this value up somewhere OTHER than the n8n_data volume (password
# manager, vault, encrypted note). Losing it makes every stored credential
# permanently unreadable, even if the volume itself is intact.
N8N_ENCRYPTION_KEY=change-me-generate-with-openssl
```
> π‘ **Note**
>
> If you are currently on an n8n set up with SQLite and want to upgrade it to PostgreSQL, you can export your current settings (workflows + credentials) to import them to the new DB (providing you have a path such as `./local-files/files` mounted):
> ```bash
> docker exec n8n n8n export:workflow --all --output=/files/workflows.json
> docker exec n8n n8n export:credentials --all --output=/files/credentials.json
> docker compose down
> # Extract the key from the SQLite instance and replace the 'N8N_ENCRYPTION_KEY' value with it in .env:
> docker run --rm -v n8n_data:/data alpine sh -c "apk add --no-cache jq >/dev/null 2>&1; jq -r .encryptionKey /data/config"
> ```
> - Then, once you have launched the PostgreSQL + n8n instance, import them:
> ```bash
> docker exec n8n n8n import:workflow --input=/files/workflows.json
> docker exec n8n n8n import:credentials --input=/files/credentials.json
> ```
```yaml
# Deploy
docker compose pull
docker compose up -d
# Check status
docker compose logs -f
```
- Try accessing it on the website:
- Complete your registration to receive a free license key (indicate that you are not using n8n for work purposes).
## Step 2 - Logic of the patching workflow
This is the most brainy part of the whole series - we will have two n8n workflows. The diagram below joins them together to showcase the entire workflow logic. The trigger is Patchmon, our source of truth regarding updates. If updates are found, an n8n workflow is triggered (see in Step 5), which resolves the host and acquires a per-group lock, optionally opens an Uptime Kuma maintenance window so the coming reboot or snapshot pause doesn't fire a false alert, then calls for a snapshot (if a Proxmox guest) by running a sub-workflow (Step 3 below).
Once a snapshot is taken, the main workflow kicks in by triggering the relevant Semaphore job template for the host's group. Yes, there is not just one template for patching; we need to take the nature of the host group into account (we configured them in Part 1 of this series). For example, with a web server running nginx, we will want to ensure that nginx is running after patching is complete.
Here is a diagram of the whole workflow logic (thanks, Claude, for putting my ideas together here):
```bash
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 0. TRIGGER β
β PatchMon webhook β n8n β
β payload: { hostname, event_type, severity, β
β host_id, pending_count, threshold } β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββ
β Hostname on the β
β EXCLUDED_HOSTNAMES β
β list? β
ββββββββββββββββββββββββββ
yes β β no
βΌ β
ββββββββββββββββββββββββ β
β SKIP - patch by hand β β
β (e.g. Proxmox hosts) β β
ββββββββββββββββββββββββ β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 1. RESOLVE HOST β
β Semaphore: run patchmon_lookup.yml, Limit=hostname β
β poll task β parse output β
β returns: groups[], is_proxmox_guest, proxmox node/type/vmid, β
β in_alert_cooldown β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββ
β Already in a 24h β
β alert cooldown? β
βββββββββββββββββββββββββ
yes β β no
βΌ β
ββββββββββββββββββββββββ β
β SKIP - already β β
β alerted, unresolved β β
ββββββββββββββββββββββββ β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 2. DECIDE β
β n8n: match groups[] against groupβtemplate rule table β
β β chosen Semaphore patch template β
β β lock_key (shared by hosts in the same group, e.g. Galera) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β Host is security-only β
β tier AND this alert is β
β a general (non-security)β
β event? β
βββββββββββββββββββββββββββ
yes β β no
βΌ β
ββββββββββββββββββββββββ β
β SKIP - security-only β β
β tier, patch by hand β β
ββββββββββββββββββββββββ β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 3. ACQUIRE LOCK (no database, no Postgres) β
β Semaphore: run patchmon_tag_host.yml, action=add β
β β’ Proxmox guest β tag "being_patched" on the Proxmox API, β
β check-then-set inside one Ansible task run β
β β’ non-guest β atomic "mkdir /run/patchmon-being-patched"β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββ
β Lock acquired? β
βββββββββββββββββββββββββ
no β β yes
βΌ β
βββββββββββββββββββββββββ β
β EXIT - another host β β
β in the lock group is β β
β already being patched β β
βββββββββββββββββββββββββ β
βΌ
ββββββββββββββββββββββββββββ
β UptimeKuma maintenance β
β enabled? (Part 3) β
ββββββββββββββββββββββββββββ
yes β β no
βΌ β
ββββββββββββββββββββββββ β
β Open Kuma maintenanceβ β
β window for this host β β
ββββββββββββββββββββββββ β
ββββββββ¬βββββββ
β
βΌ
βββββββββββββββββββββββββ
β Is a Proxmox guest? β
βββββββββββββββββββββββββ
yes β β no
βΌ β
βββββββββββββββββββββββββββββββββ β
β 4. SNAPSHOT (guests only) β β
β Semaphore: snapshot β β
β template, poll to done β β
βββββββββββββββββββββββββββββββββ β
β β
βΌ β
ββββββββββββββββββββββ β
β Snapshot succeeded?β β
ββββββββββββββββββββββ β
no β β yes β
βΌ ββββββββββ¬ββββββββ
ββββββββββββββββββ βΌ
β ABORT: alert, β ββββββββββββββββββββββββββββββββββββββββββββββββ
β release lock β β 5. PATCH β
ββββββββββββββββββ β Semaphore: run chosen template, Limit=hostβ
β poll (generous timeout, up to 45 min) β
ββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββ
β 6. VERIFY (via PatchMon's own count) β
β ansible ok + 0 pending β SUCCESS β
β ansible ok + still pending β PARTIALβ
β ansible error / timeout β FAILURE β
ββββββββββββββββββββββββββββββββββββββββ
success β partial β failure β
βΌ βΌ βΌ
ββββββββββββββ βββββββββββββββ ββββββββββββββ
β β
success β ββ οΈ partial +β β π₯ failure β
β alert β β 24h cooldownβ β alert w/ β
β β β tag + alert β β log excerptβ
ββββββββββββββ βββββββββββββββ ββββββββββββββ
β β β
βββββββββββββββ΄ββββββββββββββ
βΌ
ββββββββββββββββββββββββββββββββββββββββββ
β 7. CLEANUP (always runs once locked) β
β Semaphore: patchmon_tag_host.yml, β
β action=remove - this IS the lock β
β release (tag removed / dir deleted) β
β + remove Uptime Kuma maintenance β
β window (optional) β
ββββββββββββββββββββββββββββββββββββββββββ
```
## Step 3: Import the Poll-Semaphore Task sub-workflow
Since the workflow is quite large, it is best to split it into a parent-child relationship. If we take a step back and think about on which occasions Semaphore would be called, it's the following:
- Inventory lookup - confirm host details & tags
- Tagging - apply or remove a tag (Proxmox guests only) - `being_patched` .
- Snapshotting - create a snapshot before a patch is applied
- Patching - carry out a runbook relevant to that type of host after snapshot has completed
- Maintenance toggling *(optional, Part 3)* - open or close an Uptime Kuma maintenance window around the patch run
Import the .JSON workflow into n8n:
π [PatchMon - Poll Semaphore Sub-Task.json](attachments/PatchMon - Poll Semaphore Sub-Task.json)
### Sub-workflow explained
This sub-workflow exists so the βkick off a Semaphore task and wait for itβ logic is written once instead of four times. The parent hands it a `taskId` and a `timeoutMinutes` value that's different per call - lookups, tag add/remove, and maintenance start/stop all get 2 minutes (`POLL_TIMEOUT_MINUTES_TAG`), snapshots get 10, patches get 45 (they need the room for reboots) - and it hands back a status and the raw task output text once the Semaphore task is done or the clock runs out.
```bash
ββββββββββββββββββββββββββββββββ
β On Execute (from parent) β β Execute Workflow nodeaces
β inputs: baseUrl, projectId, β 7 types: lookup, tag-add, snapshot,
β taskId, timeoutMinutes, β patch, tag-remove,
β pollIntervalSeconds β maintenance-start and stop
ββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββ
β Set Start Time β startedAt = now()
ββββββββββββββββββββββββββββββββ
β
βββββββββββββΌβββββββββββββββ
β Wait Before Poll βββββββββββββββββββββββββββββββ
β sleeps pollIntervalSecs β β
βββββββββββββ¬βββββββββββββββ β
βΌ β
ββββββββββββββββββββββββββββββββ β
β GET Semaphore Task Status β GET /tasks/{taskId} β
ββββββββββββββββββββββββββββββββ β
β β
βΌ β
ββββββββββββββββββββββββββββββββ β
β β β
β Merge Timing Fields β elapsedMinutes = β
β β (now β startedAt) / 60s β
ββββββββββββββββββββββββββββββββ β
β β
βΌ β
ββββββββββββββββββββββββββββ β
β Finished or timed out? β β
β status β {success,error, β β
β stopped} OR elapsed β₯ β β
β timeout β β
ββββββββββββββββββββββββββββ β
no β β yes β
ββββββββββββββββΌβββββββββββββββββββββββββββββββββββ
βΌ
ββββββββββββββββββββββββββββββββ
β GET Semaphore Task Output β GET /tasks/{taskId}/raw_output
ββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββ
β Set Final Result β Status: success/error/stopped/timeout
β output: the task's β
β raw log text) β
βββββββββββββββββββββββ
β
βΌ
Returns to the parent's
Execute Workflow node
```
### Create a Semaphore API token and add it into n8n
- On your Semaphore UI instance, click on your username at the bottom left and go to API Tokens β click on the βNew tokenβ button.
- Value: `PatchMon - Poll Semaphore Task`
- Expires: Never
- Save it in your password manager (1password / Bitwarden / Vaultwarden, etc.), as you will not be able to see that token secret anymore.
### Set up a Bearer token in n8n
- Go into the imported workflow, find the tile called βGET Semaphore Task Statusβ.
- Generic Auth Type: Bearer Auth
- Bearer Auth - click on the βSet up credentialβ button
- In the new window, set up the authentication variable:
- Value: paste the token value from Semaphore
- Also set the name in the top left corner to ensure that it is not just called βBearer Authβ.
- Re-open the same tile and choose the new bearer token from the drop-down list.
- Find the other tile called βGET Semaphore Task Outputβ and apply the same header as well.
## Step 4: Set up helpers for the main workflow
For the main workflow to work, we will need to launch two playbooks - one that gets the groups, the other that deals with adding a tag when a host is being patched.
### Patchmon lookup playbook
In order to detect what are the groups that the host to be patched is a member of, we need to call a little harmless playbook in order to get the basics - no changes are made, no SSH connection is needed. Add this to your Gitea or other source version control system.
- The expected location and file name is `helpers/patchmon_lookup.yml`
π [patchmon_tag_host.yml](attachments/patchmon_tag_host.yml)
- In Semaphore, add a new template. Remember to add the Proxmox vault information to be able to reach each cluster/standalone host. Task Templates β New template.
- Name: `Patchmon Lookup Host` (or similar)
- Path to playbook file: `helpers/patchmon_lookup.yml`
- Inventory: All Sites
- Repo: your repo
- Vaults: Proxmox Vault
- Tick the box for limit, which will be supplied in the HTTP request from n8n
- Note down the task ID once you open the template.
### Patchmon Tag Playbook
The second playbook is concerned with adding and removing a tag called `being_patched` (change it to something else if you prefer). It is done in such a way (using `ansible.builtin.uri`) that will work around the situation of us having more than one set of Proxmox credentials.
- Create a file under `helpers/patchmon_tag_host.yml` .
π [patchmon_tag_host.yml](attachments/patchmon_tag_host.yml)
- Add it in Semaphore in the same way as we did it with the previous playbook.
- Remember to tick the βLimitβ box.
## Step 5: Import the main Patchmon Orchestrator Workflow
Now when we have the ID of the sub-workflow, we can proceed with importing the main one.
π [PatchMon Auto-Patch Orchestrator.json](attachments/PatchMon Auto-Patch Orchestrator.json)
### Semaphore connection & task templates
| Variable | Description |
| --- | --- |
| `SEMAPHORE_BASE_URL` | Base URL of your Semaphore instance, e.g. `http://192.168.8.26:3000`. |
| `SEMAPHORE_PROJECT_ID` | Semaphore project ID all templates below belong to. |
| `SEMAPHORE_LOOKUP_TEMPLATE_ID` | Template ID for the Lookup task (`patchmon_lookup.yml`). Resolves group membership, Proxmox identity, and alert-cooldown status. |
| `SEMAPHORE_TEMPLATE_SNAPSHOT` | Template ID for the Proxmox snapshot task (`proxmox_snapshot_host.yml`). |
| `SEMAPHORE_TEMPLATE_TAG_HOST` | Template ID for the lock acquire/release task (`patchmon_tag_host.yml`). |
| `SEMAPHORE_TEMPLATE_GALERA` | Template ID for the Galera patch playbook (`patch_debian_galera.yml`). |
| `SEMAPHORE_TEMPLATE_NGINX` | Template ID for the Nginx patch playbook (`patch_debian_nginx.yml`). |
| `SEMAPHORE_TEMPLATE_DEBIAN` | Template ID for the general Debian/Ubuntu patch playbook (`patch_debian_single_hosts.yml`). Fallback template for unmatched hosts. |
| `SEMAPHORE_TEMPLATE_UPTIMEKUMA_MAINTENANCE` | Template ID for starting/stopping an Uptime Kuma maintenance window (if you followed Part 3 of this series). |
### Alerting
| Variable | Description |
| --- | --- |
| `ALERT_WEBHOOK_URL` | Discord/Telegram webhook for success, partial, and failure alerts. Treat as a secret. |
### Polling behavior
| Variable | Description |
| --- | --- |
| `POLL_TIMEOUT_MINUTES_LOOKUP` | Minutes n8n polls a running Lookup task before timing out. |
| `POLL_TIMEOUT_MINUTES_TAG` | Same, for the tag add/remove (lock) task. |
| `POLL_TIMEOUT_MINUTES_SNAPSHOT` | Same, for the snapshot task. |
| `POLL_TIMEOUT_MINUTES_PATCH` | Same, for the patch task. Needs the most headroom: dist-upgrade plus reboot takes longest. |
| `POLL_INTERVAL_SECONDS` | How often n8n checks task status while polling. |
### Host filtering & patch scope
| Variable | Description |
| --- | --- |
| `EXCLUDED_HOSTNAMES` | Comma-separated hostnames that never trigger a patch run. |
| `SECURITY_ONLY_GROUPS` | Comma-separated inventory groups limited to security updates only. Empty: full updates everywhere. |
### Uptime Kuma integration
| Variable | Description |
| --- | --- |
| `UPTIMEKUMA_MAINTENANCE_ENABLED` | On/off switch for Uptime Kuma maintenance windows. Must be a String field ("true"/"false"), not Boolean. |
| `UPTIMEKUMA_DURATION_BUFFER_MINUTES` | Extra buffer minutes added to the estimated patch duration for the maintenance window. |
| `UPTIMEKUMA_INSTANCES` | JSON array of Uptime Kuma instances: `name` and `url` per entry. |
### Workflow explained in short
- `Normalize Trigger Payload` + `Global Config` are the setup stage. Fill in manually as explained above.
- `Is Host Excluded?` / `Skip - Host Excluded` is the hard exclude gate.
- `Start Semaphore Lookup Task` through `Parse Group Names` is host resolution. Ensure the sub-workflow is mapped there.
- `In Alert Cooldown?` is the repeat-alert guard.
- `Decide Template + Lock Key` is the one Code node you will actually maintain as your fleet grows. It's a small priority-ordered rule table mapping Ansible groups to Semaphore template IDs and lock keys.
- `Standard Update on Security-Only Host?` is the security-tier gate.
- `Start Tag-Add Task` (ensure the sub-workflow is mapped there) through `Lock Acquired?` is the locking step (Proxmox tag or local `mkdir`, decided automatically inside the playbook by whether `proxmox_vmid` is defined).
- **UptimeKuma Maintenance Enabled? (Start)** opens a maintenance window on every configured Kuma instance for this host before it's touched. Off by default; when off (or misconfigured) it routes straight through to `Is Proxmox Guest?` untouched.
- `Is Proxmox Guest?` onward splits into snapshot-then-patch for guests, or patch-only for everything else.
- `Patch Outcome` is the three-way verification switch.
- Everything funnels into `Start Tag-Remove Task `(ensure the sub-workflow is mapped there) / `Poll Tag-Remove Task `(again, check the mapping), which is both the alerting cleanup and the actual lock release.
- **UptimeKuma Maintenance Enabled? (Stop)** sits right before `Start Tag-Remove Task` - every alert path (success, partial, failure, even a failed snapshot) passes through it, so the window is always closed the same way it was opened, and a host is never left stuck "in maintenance."
### Patchmon Auto-Patch Orchestrator in an ASCII flowchart:
```bash
Incoming PatchMon Webhook [Trigger]
β
βΌ
Normalize Trigger Payload (hostname, eventType, severity, hostId, pendingCount, pendingThreshold)
β
βΌ
Global Config (every non-secret setting lives here, CE has no Variables/Environments panel)
β
βΌ
Is Host Excluded? ββyesβββΊ Skip - Host Excluded (exit)
βno
βΌ
Start Semaphore Lookup Task [sub-workflow] βββΊ Poll Lookup Task βββΊ Parse Group Names
β
βΌ
In Alert Cooldown? ββyesβββΊ Skip - In Alert Cooldown (exit)
βno
βΌ
Decide Template + Lock Key
β
βΌ
Standard Update on Security-Only Host? ββyesβββΊ Skip - Standard Update on Restricted Host (exit)
βno
βΌ
Start Tag-Add Task [sub-workflow] βββΊ Poll Tag-Add Task βββΊ Parse Tag-Add Result
β
βΌ
Lock Acquired? ββnoβββΊ Lock Busy - Exit (exit)
βyes
βΌ
UptimeKuma Maintenance Enabled? (Start) ββnoββββββββββββββββββββββββββ
βyes β
βΌ β
Start Maintenance-Start Task [sub-workflow] β
βββΊ Poll Maintenance-Start Task β
βββΊ Parse Maintenance-Start Result β
β β
βΌ β
Is Proxmox Guest? ββββββββββββββββββββββββββββββββββββββββββββββββββββ
βnoβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
βyes β
βΌ β
Start Snapshot Task [sub-workflow] βββΊ Poll Task βββΊ Snapshot Ok? β
β β
βno yesβ β
βΌ βΌ β
Snapshot Failed Alert Context Start Patch Taskβββββββ
β [sub-workflow]
β β
β βΌ
β Poll Patch Task βββΊ Parse Patch Verify Result
β β
β ββββββββββββββ
β βΌ
β Patch Outcome (3-way)
β ββββββββββββββββββ¬ββββββββββββββββββ
β successβ partialβ failureβ
β βΌ βΌ βΌ
β Send Discord/Telegram Build Partial Build Failure
β Success Message Alert Context Alert Context
β β β β
β β βΌ βΌ
β β Send Discord/Telegram Alert
β β β β
β βΌ βΌ βΌ
βββββββββββββββββββββΊβββββββββββββββββΊββββββββββββββββββΊβββββ
ββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
UptimeKuma Maintenance Enabled? (Stop) ββnoβββββββββββββββββββ
βyes β
βΌ β
Start Maintenance-Stop Task [sub-workflow] β
βββΊ Poll Maintenance-Stop Task β
βββΊ Parse Maintenance-Stop Result β
β β
βΌ β
Start Tag-Remove Task ββββββββββββββββββββββββββββββ
[sub-workflow] βββΊ Poll Tag-Remove Task
β
βΌ
(Exit - releases the lock either way)
```
Before we can really run this workflow, we will need to configure Patchmon to send out webhooks when new updates are available. So run the test trigger for the workflow and head to Patchmon.
## Step 6: Finalize the connection with Patchmon
The whole workflow starts with Patchmon. It is this very system that serves as a source of truth in terms of what updates are pending, whether standard or security updates. We will need to configure it to send notifications to n8n, which will then trigger patching jobs with Semaphore UI. Then, we will trigger a re-check by a Patchmon agent and send alerts via Discord, if required.
### Create an Alert Channel
Since we now have a webhook created in n8n, we can link it in Patchmon.
- Go to Reporting β Click on the βAdd Destinationβ button and select βWebhookβ.
- Test the webhook:
- Go to n8n and click on βExecute Workflowβ
- Click on the βTestβ button in Patchmon to fire a βTestβ button while still in the Destinations section.
> π‘ **Note**
>
> Note: n8nβs **test **webhook differs from the **production **one - the test one is executed once when being actively triggered (waiting for a webhook to come), whereas the production webhook will trigger even when you do not actively press the βExecuteβ button. For the production trigger to work properly, always publish your workflow to ensure you are not triggering on an older version of your workflow.
### Configure a route for the webhook
With the webhook created, go to βEvent Rulesβ sub-section (within Reporting) and click on βAdd event ruleβ. Select these two events:
- Pending updates threshold exceeded
- Security updates threshold exceeded
- Minimum severity: `Informational`
### Configure thresholds
I recommend taking a quick look at the [**manual related to Patchmonβs thresholds**](https://patchmon.net/docs/patchmon-admin-guide). Patchmon needs to know on a per-host basis what is the acceptable threshold of accumulated updates before it fires the n8n workflow to apply a patch.
- Firstly, go to Reporting β Alert Lifecycle β scroll down to βAlerts systemβ. Make sure that the toggle for master switch is on.
- Once on, scroll down and toggle the following two to enable them:
- Host Security Updates Exceeded
- Host Pending Updates Exceeded (consider covering these later once you run a few tests with the security updates)
- Severity: Warning
- Regarding the threshold, I recommend finding a host with that is the most out of date and setting the threshold one below that, so that during testing, you will not have too many webhooks coming into n8n.
- Later on once tested, the recommended value of 1 for security updates and 10 for standard updates is fine.
> π‘ **Note**
>
> Just be careful about not setting the limit for updates too low, or you may a large number of hosts to update, which may clog up your IO resources.
> 
## Step 7 - Run, run, auto-patching workflow!
Getting it work initially will likely require a number of attempts. In my experience, it was best to configure updates coming out for just one host. Taking a manual snapshot of it is advisable before you verify that the workflow-driven one works.
- My most out of date host was `mail1` - look at how badly outdated this iRedMail host was, it is almost embarrassing to share!
- Once the workflow kicks in (ensure you clicked through every tile and provided the required variables in the Config tile), the first job that will be triggered in Semaphore is the lookup task.
- If all good and it is a Proxmox guest (VM or an LXC), a snapshot is taken. In your Proxmox UI, observe that a snapshot was, in fact, taken, by the tag being added:
- Then the `Decide Template + Lock Key` is loaded - you may need to configure playbooks as per your type of servers
- This is then followed by the actual patching job being called in Semaphore, as per your configured playbooks and then polled for the result. The playbook forces a re-check via the Patchmon agent at the end.
- And lastly, a nice Discord or Telegram message π What more can you want?
- Now letβs check the status in Patchmon!
### Troubleshooting your patching job executions
Lots of things can and will go wrong, so letβs look into a few:
- **No workflows coming to n8n** - check the Delivery log section (`your_serever:port/reporting?tab=alerts`) to see if a webhook even went out. If yes and it errored out, check what the error is. Error 404 indicates the webhook address is incorrect or there was nothing listening on that (when using the test hook, you need to click on the βExecuteβ button).
- **Error when taking a snapshot** related to Proxmox - ensure that the credentials saved in Semaphore (such as via Ansible Vault) are correct. If needed, update them in Semaphore or re-create the Ansible Vault and then reflect it in the `group_vars/` folder.
Once you are confident with the process, you can decrease the threshold for updates in Patchmon to ensure that your fleet gets patched.
Since this is quite a complex workflow, do let me know if you identify any bugs or make improvements that would be worth sharing with others.
## Step 8 - Patch Security updates only
Let's say that you have a group of hosts called **webservers **and you want them to receive security updates only. This group was already created in Part 1 of this series - the web1 and web2 VMs are tagged as βwebβ.
- Let's add βweb1β and βweb2β into a group called βwebserversβ in terms of their Proxmox tags.
- At the start, βweb1β had a number of pending updates waiting, both security and standard, as shown below:
- The result was that only the security updates got applied!
- Verify the finding in Patchmon:
This is pretty awesome! You can consider applying this for your Proxmox hosts and other critical services that you want to keep up to date in terms of security, but would otherwise prefer to update manually.
## Conclusion
The Part 4 of this series closes the loop: **PatchMon **spots the drift, **n8n **decides whether and how to act on it, **Semaphore** does the actual work, and **Uptime Kuma** keeps your alerting quiet while it happens and the monitors the hosts afterwards, in case something happens post-patching.
We have got a fleet that patches itself, snapshots itself first, verifies its own work, and tells us about it on Discord or Telegram - no database of its own anywhere in the loop, just Proxmox tags and a marker file standing in for locks.
A few things worth doing before you point this at production:
- Put these instances (n8n, Semaphore, Patchmon, Kuma) behind a **reverse proxy** running on **HTTPS only**.
- Implement **firewall rules **on these hosts (`iptables`, `nftables`, `ufw` or similar) that allows connection only from your server subnet.
- Add **fail2ban** on each of these hosts, allowing only certain endpoints for SSH access.
- The workflow is deliberately stateless, no lock table, no external queue. That may work well in a home environment. Originally, I prepared this tutorial by saving all information in a `postgresql` database, which helps with retrieval and locks. However, I wanted to simplify the process and not add another system to manage as part of the process. On a larger scale, however, it would likely be the right approach.
Did you find a bug, or improve on something here? Let me know in the comments! I would rather fix it for everyone than to have five people independently trip over the same edge case. Knowledge is to share - primarily with other humans, then our fellow AI friends :)