Proxmox

Pre-requisites:

• Each node is on the same version.
• They can talk to each other on the local network. If possible, connect them directly, the guide below will expand on how.
• They can resolve each others hostnames (so that we do not need to rely on IP addresses, as the API communication requires a certificate, which needs to be issued to a valid subject (not an IP address).

Connect the Proxmox nodes via a direct physical line:

• Let’s say that each Proxmox node has an available physical interface via which they can communicate with each other. In the case below, it is en3s0, i.e. the third port on the device.
• Instead of relying on LAN communication via a router, we can connect them directly. This way, even if the LAN comms fails, they can still talk to each other regardless of what happens on the network.
• Firstly, have a look at your Proxmox node to see what interface is available. You might need to free one up to use it for this dedicated line. In the case below, the physical ports 3-6 are used for LAN traffic. We will need to remove the port no.3 to free it for another virtual interface.

• Under the node (proxmox1) → System → Network, click on the ‘Create’ button and select ‘Linux bridge’.
  • Provide the IPv4/CIDR information that applies to the specific node. Since we just need two nodes, we can use the /30 subnet that allows for us to 2 hosts (since one is the network address and the other the broadcast address). If you plan to add more nodes, you can set up the subnet to something larger, such as /29 (up to 6 usable hosts) or /28, (up to 16 usable hosts).
  • Write down the name of the physical interface that you decided to use beforehand. In this case, it is enp3s0, i.e. the third NIC on the device.

• Do the same on the second node with the difference that the IPv4/CIDR is different in the fourth octet, i.e. 10.0.0.2/30. The port number can be different, although having a mirrored set up is easier in the long-run.

Ensure that they can talk to each other

• Check that the nodes have proper names and can reach each other. See this article on how to rename a node.
• From the shell of each node, add the connection in the /etc/hosts file:

jan@proxmox1:~$ sudo nano /etc/hosts

# Add these two lines (adjust it to the IPs and names that you use):
10.0.0.1 proxmox1
10.0.0.2 proxmox2

• Do the same on the second node.
• After saving it, run a test from each:

jan@proxmox1:~$ ping proxmox2
PING proxmox2.bakalar.priv (10.0.0.2) 56(84) bytes of data.
64 bytes from proxmox2 (10.0.0.2): icmp_seq=1 ttl=64 time=1.06 ms

jan@proxmox2:~$ ping proxmox1
PING proxmox1.bakalar.priv (10.0.0.1) 56(84) bytes of data.
64 bytes from proxmox1 (10.0.0.1): icmp_seq=1 ttl=64 time=0.260 ms

Create a PVE cluster

• On version 8.x, initiate a cluster with the first node as its only member. In this case, I call the cluster name as ‘home’ – feel free to change the name:

sudo pvecm create home

• You can then review the corosync.config file to see what it produced:

jan@proxmox1:~$ sudo nano /etc/pve/corosync.conf

• In case a different IP was used because the proxmox nodes are inter-connected via another interface, you can change the IP address to match the one that you dedicated under the ‘ring0_addr’.
• Then connect to the second node and run the following:

jan@proxmox2:~$ sudo pvecm add proxmox1

• Review the output. We can see that:
  • You need to log in with the first node’s root password.
  • Accept the connection.
  • If you have them connected via more than just that one link, you might end up with a different IP address than expected.

• Review the /etc/pve/corosync.conf file one more time:

jan@proxmox1:~$ sudo nano /etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: proxmox1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.0.0.1
  }
  node {
    name: proxmox2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.0.2
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: home
  config_version: 2
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

• Troubleshooting: In case you struggle, you can also run this command to see more details about how each node is configured: corosync-cfgtool -s

BONUS: Add a Rapsberry PI quorum member to prevent split-brain scenarios

Corosync works on a quorum-based system where majority is needed. In the ideal case, there would be three nodes. We can utilize a low-powered device such as a Raspberry Pi as a quorum-only member without actually having to run Proxmox on it.

• On each Proxmox node, install:

apt install corosync-qnetd corosync-qdevice

• On a Rapsberry PI (Debian-based), install:

sudo apt install corosync-qnetd
sudo systemctl enable corosync-qnetd
sudo systemctl start corosync-qnetd

• Then on either node on the proxmox cluster, run:

pvecm qdevice setup <raspberrypi-hostname-or-ip>

• Lastly, run pvecm status to see if the third member is in there:

In case you run Proxmox and realize that you would like to change the hostname, for example when about to join two nodes together into a cluster, you can proceed in the following way:

• Run ‘hostnamectl’ to see what you have set at the moment.
• Rename it in the following files:

/etc/hostname
/etc/hosts
/etc/postfix/main.cf

• Then rename it in active by running ‘hostnamectl set-hostname your-node-name’
• Verify that it is set correctly by running ‘hostnamectl’
• Restart these proxmox-related services:

systemctl restart pvedaemon
systemctl restart pvestatd
systemctl restart pveproxy

• In case you open your node’s web interface at this point, you will notice that the old node is still there with its VMs in an unknown state. You will need to move files from one pve to another. Let’s say we renamed it from ‘node1’ to ‘proxmox1’. You would need to run the following:

# Modify these lines to your own hostnames! 
# From node1 to proxmox1:
mv /etc/pve/nodes/node1/lxc /etc/pve/nodes/proxmox1/lxc
mv /etc/pve/nodes/node1/qemu-server /etc/pve/nodes/proxmox1/qemu-server

The above will move information about all the containers and VMs onto the new node name. It is recommended to do a reboot of the node afterwards.