What to do when facing a Quorum Loss
What are the most common scenarios when a quorum is lost and thus the Galera cluster fails?
- Case 1: No Arbitrator and half of the nodes go down: Imagine a case where you do not have an Arbitrator (as mentioned in the previous chapter when we use the Raspberry Pi for it) and one of the two Proxmox nodes where we run the Galera containers goes down. Then we would not have a majority of votes and the cluster would stop serving traffic.
- Case 2: Power outage – all nodes go down: This can also happen (or more like will happen at some point whether you like it or not). When you start them up, you will notice that the cluster is not working anyway!
In either case, the cluster has to be recovered manually to prevent data loss. So it is a feature, not a bug! Please refer to this article if you have been affected by the cluster shutdown (or are just simulating it).
In a nutshell, the idea is to SSH into each node and verify which one is safe to bootstrap (the result should be 1 on one of them), then you can start a new cluster on that node. Once it is back up, you will need to restart MariaDB on each cluster node.
cat /var/lib/mysql/grastate.dat | grep safe_to_bootstrap # Run this on whichever node results in 1: sudo galera_new_cluster # Watch the logs for changes tail -n 50 /var/log/mysql/error.log # Restart Maria DB on OTHER nodes sudo systemctl restart mariadb # Verify the cluster size afterwards from any node mysql -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
Self-healing options
- Automation would be the key here. Firstly, we would run an Ansible playbook that connects to each node, verifies which node went down the last and attempts to recover it, restarting each MariaDB instance or even the entire container.
- If that fails, we could use a tool like Terraform to start new MariaDB instances from our container template and then Ansible to trigger a playbook to run updates on each container, connect them into a cluster and to check if it can extract the dumps from a node and if not, to download them from a last known backup and import them.
- We would also need to automatically update items in the back-end pool in our Load Balancer (such as by leveraging OPNSense API for HAProxy).
Such an approach definitely warrants its own article – let me know in the comments below if you are interested!
This concludes our rather comprehensive guide on how to deploy a Galera Cluster using technologies like Proxmox, OPNSense, HAProxy and LXC containers. Feel free to share your own experience and stack, perhaps this guide could be expanded to account for more options in the set up.