3. Prepare a Script of Scripts for automated shutdown of Proxmox hosts

Originally I prepared a script to manage 1 or 2 hosts to gracefully shutdown when a battery reaches a certain level that you would define in a variable. Then I realized – what if you have more? The script below accounts for that:

sudo nano /usr/local/sbin/ups_manager.sh

#!/bin/bash

#================================================
#             UPS SHUTDOWN MANAGER              #
#================================================

# --- User Configuration ---
EMAIL_TO="[email protected]"
PROXMOX_HOSTS=(
    "1.2.3.4"  # Proxmox2 (secondary to shut down first)
    "4.3.2.1"  # Proxmox1 (primary to shut down last)
)
START_PERCENT=80
END_PERCENT=30
PI_SHUTDOWN_PERCENT=12

# --- System Configuration ---
LOG_FILE="/var/log/ups_manager.log"
FLAG_DIR="/tmp/ups_shutdown_flags"
PI_FLAG_FILE="${FLAG_DIR}/pi_shutdown.flag"
SSH_KEY_FILE="/root/.ssh/upsmanage_rsa"

# --- Cron-safe paths ---
APCACCESS_CMD="/usr/sbin/apcaccess"
GREP_CMD="/usr/bin/grep"
AWK_CMD="/usr/bin/awk"
BC_CMD="/usr/bin/bc"
SSH_CMD="/usr/bin/ssh"

#================================================
#                  FUNCTIONS                    #
#================================================

log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}

send_email() {
    local subject="$1"
    local body="$2"
    echo "$body" | mutt -s "$subject" "$EMAIL_TO"
    log_message "Email sent to $EMAIL_TO: $subject"
}

graceful_shutdown_host() {
    local HOST_IP=$1
    log_message "Attempting graceful (non-blocking) shutdown of $HOST_IP..."
    
    # This command runs remotely on the Proxmox host
    $SSH_CMD -i $SSH_KEY_FILE -o ConnectTimeout=10 root@$HOST_IP '
        log_msg() { echo "$(date): $1"; }
        
        log_msg "Received shutdown signal from UPS manager."
        
        # 1. Gracefully shut down all running VMs
        log_msg "Sending shutdown signal to all QEMU VMs..."
        for vmid in $(qm list | grep running | awk "{print \\$1}"); do
            qm shutdown $vmid
        done
        
        # 2. Wait 5 minutes (300 seconds) for VMs to shut down
        log_msg "Waiting 300 seconds for graceful VM shutdown..."
        sleep 300
        
        # 3. Forcefully stop any VMs still running (like stuck Windows VMs)
        log_msg "Forcing shutdown of any remaining VMs..."
        for vmid in $(qm list | grep running | awk "{print \\$1}"); do
            log_msg "VM $vmid is stuck. Forcing stop."
            qm stop $vmid
        done
        
        # 4. Gracefully shut down all running containers
        log_msg "Sending shutdown signal to all LXC Containers..."
        for ctid in $(pct list | grep running | awk "{print \\$1}"); do
            pct shutdown $ctid
        done
        
        # 5. Wait 2 minutes (120 seconds) for containers
        log_msg "Waiting 120 seconds for containers to stop..."
        sleep 120
        
        # 6. Forcefully stop any containers still running
        log_msg "Forcing shutdown of any remaining containers..."
        for ctid in $(pct list | grep running | awk "{print \\$1}"); do
            log_msg "Container $ctid is stuck. Forcing stop."
            pct stop $ctid
        done
        sleep 60
        
        # 7. Shut down the Proxmox host
        log_msg "All guests stopped. Shutting down Proxmox host now."
        shutdown -h now
    '
}

#================================================
#                SCRIPT LOGIC                   #
#================================================

mkdir -p "$FLAG_DIR"
log_message "Script started. Checking UPS status..."

# --- Get UPS Status (using full paths) ---
APC_OUTPUT=$($APCACCESS_CMD)
if [ $? -ne 0 ]; then
    log_message "FATAL: 'apcaccess' command failed. Is apcupsd running?"
    exit 1
fi

UPS_STATUS=$(echo "$APC_OUTPUT" | $GREP_CMD "STATUS" | $AWK_CMD '{print $3}')
# Use BCHARGE per your discovery
BATT_PERCENT=$(echo "$APC_OUTPUT" | $GREP_CMD "BCHARGE" | $AWK_CMD '{print $3}' | cut -d'.' -f1)

if [ "$UPS_STATUS" != "ONBATT" ]; then
    log_message "UPS is on line power ($UPS_STATUS). No action needed."
    exit 0
fi

log_message "WARNING: UPS is on battery! Current level: ${BATT_PERCENT}%"

# --- Dynamic Threshold Calculation ---
declare -a THRESHOLDS
HOST_COUNT=${#PROXMOX_HOSTS[@]}

if [ "$HOST_COUNT" -eq 1 ]; then
    THRESHOLDS=($START_PERCENT)
else
    RANGE=$(($START_PERCENT - $END_PERCENT))
    INTERVALS=$(($HOST_COUNT - 1))
    STEP=$(echo "scale=4; $RANGE / $INTERVALS" | $BC_CMD)
    
    for (( i=0; i<$HOST_COUNT; i++ )); do
        THRESH=$(echo "scale=4; $START_PERCENT - ($i * $STEP)" | $BC_CMD)
        THRESHOLDS[$i]=$(printf "%.0f" "$THRESH")
    done
fi

log_message "Calculated shutdown thresholds: ${THRESHOLDS[*]}"

# --- Check Proxmox Hosts ---
for (( i=0; i<${#PROXMOX_HOSTS[@]}; i++ )); do
    HOST_IP=${PROXMOX_HOSTS[$i]}
    HOST_THRESHOLD=${THRESHOLDS[$i]}
    FLAG_FILE="${FLAG_DIR}/host_${HOST_IP}.flag"
    
    if [ "$BATT_PERCENT" -le "$HOST_THRESHOLD" ] && [ ! -f "$FLAG_FILE" ]; then
        log_message "TRIGGER: Battery at ${BATT_PERCENT}%. Threshold of ${HOST_THRESHOLD}% met for ${HOST_IP}."
        touch "$FLAG_FILE"
        
        SUBJECT="UPS ALERT: Shutting down Proxmox Host ${HOST_IP}"
        BODY="UPS battery level reached ${BATT_PERCENT}%.

Triggering graceful (non-blocking) shutdown for Proxmox host at ${HOST_IP} (Threshold: ${HOST_THRESHOLD}%)."
        send_email "$SUBJECT" "$BODY"
        
        # Call shutdown function in the background
        graceful_shutdown_host "$HOST_IP" &
        
    elif [ -f "$FLAG_FILE" ]; then
        log_message "INFO: Shutdown command for ${HOST_IP} already sent."
    fi
done

# --- Check Raspberry Pi Self-Shutdown ---
if [ "$BATT_PERCENT" -le "$PI_SHUTDOWN_PERCENT" ] && [ ! -f "$PI_FLAG_FILE" ]; then
    log_message "CRITICAL: Battery at ${BATT_PERCENT}%. Shutting down myself."
    touch "$PI_FLAG_FILE"
    sudo shutdown -h now
fi

log_message "The script has finished."

• Feel free to execute it by running bash /usr/local/sbin/ups_manager.sh .
• When connected to power, you will simply get a message that UPS is on line power (ONLINE). No action needed..

---

Script Explained – in words

It’s designed to be run by cron every minute on your Raspberry Pi. Its sole job is to check the UPS status and decide if any action is needed.

Here is a step-by-step breakdown of its logic:

• Check Power Status: The script first checks if the UPS is ONBATT (on battery). If the status is ONLINE (on mains power), it logs a simple ‘all good’ message and immediately exits. No further action is taken.
• Calculate Dynamic Thresholds: If the UPS is on battery, the script’s first action is to calculate a unique shutdown threshold for every host in your PROXMOX_HOSTS array.
  • It takes your START_PERCENT (e.g., 80) and END_PERCENT (e.g., 30).
  • It “spreads” the shutdowns evenly across this range. For example, with two hosts, the thresholds are [80, 30]. If you added a third host, the script would automatically calculate [80, 55, 30]. This ensures all your servers don’t try to shut down at once.
• Check Each Host: The script loops through your PROXMOX_HOSTS array, checking each host one by one. For each host, it asks two questions:
  • Question 1: Is the current battery level (BCHARGE) at or below this host’s unique threshold?
  • Question 2: Has a “shutdown flag” file already been created for this host? (This prevents sending the shutdown command many times).
• Trigger the Shutdown: If the answer is “Yes” to Q1 (battery is low) and “No” to Q2 (no flag exists), the script triggers the shutdown:
  • It creates the flag file (e.g., /tmp/ups_shutdown_flags/host_1.2.3.4.flag) to “lock” this host.
  • It sends you a detailed email alert.
  • It calls the graceful_shutdown_host function in the background (using &). This is important: the Pi sends the command but does not wait for it to finish. This allows the script to finish quickly so it can check on other hosts or the Pi itself.
• Final Failsafe (Pi Shutdown): After checking all Proxmox hosts, the script does one last check for itself. If the battery is at or below the PI_SHUTDOWN_PERCENT (e.g., 12%), it shuts itself down.

---

Script Explained – in a flow chart

• You can access it live on this this link (no login required).

So what now? We will need to make it executable and add it to cron. Yet before we do that, let’s configure the other two scripts that are triggered automatically when the UPS switches to battery power and then back to mains.

Go back to course overview: Automate Graceful Shutdown & Wake on LAN for Proxmox with RPi (APC)