# How to join 2 OPNSense boxes in a cluster (pfSync) with LAN + WAN High Availability (CARP) [TOC] ## Sources: - [OPNSense documentation](https://docs.opnsense.org/manual/how-tos/carp.html) ## Pre-requisites: - 2x OPNSense boxes - we will set them up via CARP as main and backup for each interface (WAN1, WAN2, LAN and SYNC). - Both boxes need to be on the same version. If they are virtualized, you can clone one (if you do, [a] disable the LAN interface of the cloned VM, [b] boot it up and disable DHCP on the backup later until they are in CARP where only one serves DHCP traffic, [c] change the hostname so that it is unique and [d] change IPs of the static interfaces that you use.) - Each OPNSense box will need at least 3x network ports: - WAN1 - to connect to your ISP1 - WAN2 (optional) - for your ISP2 - SYNC - a dedicated line to talk to each other on deciding who is main / backup. Optional but highly recommended to wire them directly together. - LAN - to connect to a switch or access point for additional devices to connect - 1-2x WAN routers - preferably different last mile providers - You do not need a static IP address on either, we can work with dynamic just fine, even if you plan to serve websites and other services (such as a Minecraft server, mail server, etc.) - however, the ISP must not block the ports and allow routing. - Ability to configure the ISP Router(s) to ensure they are in router mode rather than a bridge mode, so that you can connect each OPNSense device to it on a different ethernet port (the guide below helps you to configure it). --- ## Diagram:

--- ## Configure the SYNC interface - As explained in the previous step, the SYNC interface is optional but highly recommended to not lose connectivity during broadcast storms and other traffic that may prevent communication between the two or more nodes. Once this direct connection is established, we can run pfSync (and nothing else) on it. With pfSync, the states will be synced in real time, so that in case of a fail-over (using CARP), traffic will remain preserved. This is why we are setting up the SYNC interface and then pfSync first before CARP. - Start by connecting the OPNSense boxes together via a dedicated ethernet cable. - Create a new interface on each OPNSense under ‘Interfaces’ -> ‘Assignments’. Choose the correct MAC address of the interface.

- Under ‘Interfaces’ → ‘SYNC’ (or whatever name you chose for your interface, assign a static IP on each OPNSense box:

- You might ask why /29? This is because /30 is very restrictive, it allows for only 2 usable hosts. What if you want to add another device in the future as another backup? So the next best option is /29, which allows for 6 usable hosts. - Set up the following required firewall rules - while they are connected to each, the default firewall rule is to deny all. So even for them to be able to ping each other, we will need to set up the required firewall rules. ### Set up firewall rules - On each OPNSense box, go to ‘Firewall’ → ‘Rules’ → ‘SYNC’ and create a new rule. - **Rule no.1: Allow ICMP on the firewall to test for pings**: - Action: Pass - Interface: SYNC - Direction: in - TCP/IP: IPv4 - Protocol: ICMP - Source: Any - Destination: Any (or just SYNC net if you do not need to ping them from your LAN). - Destination ports: Any

- **Rule no.2: pfSync - to sync firewall states with each other**: - Action: Pass - Interface: SYNC - Direction: in - TCP/IP: IPv4 - Protocol: PFSYNC - Source: SYNC net - Destination: This Firewall - Destination ports: any

- **Rule no.3: XMLRPC (WebGUI access to copy configs from main to backup)** - Action: Pass - Interface: SYNC - Direction: in - TCP/IP: IPv4 - Protocol: TCP - Source: SYNC net - Destination: This Firewall - Destination port range: from 443 to 443 - Important note: If your OPNsense web admin portal is using an alternative port because you use port 443 for other services such as HAProxy, then check the steps below under ‘Troubleshooting’ to remedy that first before proceeding.

- Test that you can ping from one OPNSense box to another. - From OPNSense2:

- From OPNSense1:

### Troubleshooting - On virtualized OPNSense boxes, esp. via hypervisors like Proxmox, ensure that the interface firewall is off.

- On virtualized OPNSense boxes, in case pinging does not work and you get an error called ‘ping: sendto: Host is down’ , you will need to temporarily remove the network card on Proxmox under the VM’s hardware settings (save its MAC address) and re-attach a new one (with the same MAC address). Then remember to re-assign it under ‘Interfaces’ → ‘Assignments’. - No pings getting through? Make sure that the instances are interconnected via an ethernet cable. Double/triple check the firewall rules, inc. direction (in), protocol (TCP), source (SYNC net unless using LAN without a dedicated interface) and destination (This Firewall, not SYNC net or SYNC address!). Let's configure pfSync and XML-RPC to sync config and states before diving into CARP. ## Set up pfsync + XML-RPC - **pfsync**: This is used for synchronizing the **state tables** between two OPNsense boxes. It ensures that active network connections and their states (e.g., a web session, an ongoing download, or a VPN tunnel) are replicated from the active to the standby firewall. This means that during a failover event, established connections don't drop, providing a seamless transition for users. - It is for this service (not for CARP) that we use the dedicated SYNC line, if possible. While it is not mandatory, in case there is a broadcasting storm or other undesired traffic on the LAN, we may run into a situation where the main unit cannot be determined due to packet loss and face a split brain scenario. - **XML-RPC **(Configuration Sync): uses HTTPS (TCP/443, not flexible to change). It syncs settings like firewall rules, aliases, NAT, users, etc. If you keep each box synced manually then you do not need to use this service, but it is recommended, as otherwise you might run into a situation where you configure new firewall rules or settings on the ‘main’ unit and forget to do the same on the ‘backup unit(s)’. ### Set up pfSync and XML-RPC - Enable pfsync for firewall state syncing. On the main unit (not backup!), go to ‘System’ → ‘High Availability’ → ‘Settings’ and choose the following: - Leave the first two items unticked - Synchronize all states via: SYNC - Sync compatibility: 24.7 or above - Synchronize Peer IP: the backup unit’s IP address - Synchronize Config: the backup unit’s IP address (again) - Remote System username: root (you can set up a specific user for this purpose if you prefer) - Remote System password: - Services: choose all BUT ‘System tunables’ - Then click on the ‘Apply’ button, followed by the ‘Perform synchronization’ button.

- As soon as you click on the sync button, you should see something similar as on the screenshot below: - Click on the ‘Synchronize config to backup’ button to push the settings from the main to the backup unit to keep them consistent.

- Perform the High Availability (pfsync only) set up on your backup unit. - Just make sure to provide the correct IP address of the main unit, login details and do not fill in the IP address for the ‘Synchronize Config’ section, as we assume the main unit will always be the most up to date with config changes. - It is recommended to tick the 'Disable preempt' box in case you would like this unit to go back to backup in the event it becomes main for some time during the main unit's outage.

### Troubleshooting - Double/triple check your firewall rules for pfSync and XML-RPC, esp. the source and destination. No ports are to be specified for those services. - Make sure that if you configured ICMP firewall rules on each OPNSense box, that they can ping each other. If they cannot ping each other despite fw rules, then some other firewall is in place. For example, in a virtualized environment (such as Proxmox), ensure that firewall is not applied on that interface and that there is no IP set up on the SYNC or WAN interfaces, since OPNSense is handling that. - If you run the WebGUI in a different port than 443, then the question is what else is using that port - for example, the HAProxy plugin for OPNSense. - If unsure about which service is using the 443 interface on your OPSense box (if not the WebGUI), you can run ‘`sockstat -l | grep :443`’. - If you do run HAProxy on port 443 and the WebGUI on another, you will need to change it - rather configure HAProxy to run on another port (such as 8443 or 4443). - You will then need to change your firewall rules that forward HTTPS traffic from each WAN interface to ‘This Firewall’ to use that port (instead of 443 to use the custom one, such as 4443 or 8443 or whatever you use). - On your WAN router, change the port forwarding from external 443 to internal 8443 (or 4443, whatever you decide to use). - Lastly, you can change the WebGUI port back to 443 - go to ‘Settings’ → ‘System’ → ‘Administration’ and change it there. - Issues with XML-RPC? On either unit, run ‘cat /var/log/system/latest.log | grep xmlrpc’ to see what the logs indicate. - If XML-RPC is still not working, try connecting to the Web interface of each OPNsense by using the SYNC IP of each. Just make sure that the SYNC interface on each OPNsense box is also selected via ‘Settings’ - ‘System’ - ‘Administration’. If still not accessible, check that you have no NAT rule that could be interfering. - Unsure if pfsync is running and in which direction? Run ‘pfctl -s state | grep 20.0.0’ from either OPSense box:

--- ## Configure CARP on LAN & WAN interfaces - **CARP **(Common Address Redundancy Protocol): It is used to allow multiple devices to share a single IP address. It operates on the IP protocol 112, which is derived from OpenBSD. - **advbase**: The base internal at which CARP advertisements are sent. Should be the same on each CARP virtual line across OPNSense boxes. If it differs then you may experience sync flapping (regular undesired fail-overs). - **advskew**: A fractional delay (1/256 seconds) that helps to indicate which unit should be the main one (’master’) once back online. The value is between 1 and 100. - In a failover scenario, the ‘backup’ unit takes over the IP address if the primary device fails. This ensures continuous availability of services. The communication in the form of a regular heartbeat takes place from the interface towards the firewall and typically needs to be explicitly allowed by a firewall rule. - What are we trying to achieve here with CARP? - On the WAN interface(s), we will want the WAN router to forward traffic to just one IP address on which either OPNSense will be listening (only one is ‘main’, any other is ‘backup’ - if we have more units, then one is ‘main’ and all others are ‘backups’). For example, if on your WAN interface, your OPNSense1 box has a statically assigned IP of 192.168.70.10 and OPNSense2 box uses 192.168.70.11, then the WAN router would be forwarding web traffic (HTTP and HTTPS) to a shared interface of 192.168.70.254, for example. - On the LAN interface, we would want the DHCP (and likely also DNS) server to be the shared IP address for your hosts, as there can be only one gateway. For example, if your OPNsense1 host runs on 192.168.1.1 and the second one on 192.168.1.2, the shared interface can be 192.168.1.254. - CARP traffic is always handled on the interface that we are ‘carping’, i.e. the WAN and LAN interfaces, never on the SYNC interface, which is used only for pfSync. - To set up CARP, let us firstly set up firewall rules to allow CARP traffic on the respective interfaces. Go to ‘Firewall’ → ‘Rules’ → ‘WAN’ interface and click on the + sign. - WAN CARP Rule - Action: Pass - Interface: WAN1 - Direction: in - TCP/IP: IPv4 - Protocol: CARP - Source: WAN1 net - Destination: This Firewall - Destination port: any

- Set up the exact same rule on your other OPNSense (backup) box (unless you have them syncing via XML-RPC already and thus the config can be synced over). - Then, since we are on it, let us do the very same thing for the LAN interface. Just switch into ‘Firewall’ → ‘Rules’ → ‘LAN’ → click on the + sign. - LAN CARP Rule: - Action: Pass - Interface: LAN - Direction: in - TCP/IP: IPv4 - Protocol: CARP - Source: LAN net - Destination: This Firewall - Destination port: any

- You might ask why to create this rule if, usually, all traffic is already allowed on the LAN? Well, just in case. I also recommend increasing the priority for this rule on both interfaces. Why? Well, in case you create a rule later on that would trump this one and accidentally forward the traffic elsewhere.

- Now let us create the actual virtual IPs for each interface. Let us start with LAN. On the main OPNSense unit, go to ‘Interfaces’ → ‘Virtual IPs’ → ‘Settings’ and create a new rule. Click on the ‘advance mode’ to expand the options. - Mode: CARP - Interface: LAN - Network address: your LAN / subnet, such as 192.168.1.254/24 - Peer (ipv4 + ipv6): Leave blank for default. - Password: A unique password for this virtual IP line. You will use the same on other OPNSense box(es). - VHID Group: if you do not have any yet, then 1. Has to be unique for each virtual IP line. - advbase: 1 (= every one second) - advskew: 1 for primary and 50-80 recommended for the backup unit. If you have more backups, then spread it more like 1 (primary) - 30 (secondary) - 50 (tertiary).

- Do the same on the backup unit(s), just increase the advskew values to give the main unit higher priority, so that if it comes back online after a fail-over, the original backup would go back to being a backup.

- Then go to the ‘Virtual IPs’ → ‘Status’ section to see if the main unit is really ‘Master’ and the secondary unit is ‘Backup’. If not, on the backup unit, click on the ‘Temporarily disable CARP’ and then re-enable it to force a re-sync.

- In case they are both ‘Master’, then likely, CARP communication on the interface is being blocked. Check the rules. - Once happy with it, switch gateway + local DNS resolver information on any devices (virtual servers and containers) that have a statically assigned gateway + on your DHCP. - On OPNSense, go to ‘Services’ → ‘ISC DHCPv4’ → [LAN] → change ‘Gateway’ to the virtual IP.

- Now your devices connect to the virtual IP address and no longer to the specific OPNSense box! They do not need to know which one is active at that time, which is the beauty of this abstraction layer. - Let’s do the same for the WAN interface. On the main OPNSense unit, go to ‘Interfaces’ → ‘Virtual IPs’ → ‘Settings’ and create a new rule. - Mode: CARP - Interface: WAN1 - Network address: your WAN / subnet, such as 192.168.10.254/24 - Peer (ipv4 + ipv6): Leave blank for default. - Password: A unique password for this virtual IP line. You will use the same on other OPNSense box(es). - VHID Group: if you do not have any yet, then 2. Has to be unique for each virtual IP line. - advbase: 3 (= every three seconds) - advskew: 1 for primary and 50-80 recommended for the backup unit. If you have more backups, then spread it more like 1 (primary) - 30 (secondary) - 50 (tertiary).

- Again, to the same on the backup unit(s) with a higher advskew value and then check the ‘Status’ page.

- In case you are struggling to make them sync (e.g. both appear as ‘Master’), it could be that the WAN router has its own firewall on or is not allowing the traffic. Also, ensure that traffic is not blocked on ‘private networks’ under ‘Interfaces’ → ‘WAN’ interface on all your OPNSense boxes.

- At this point it is important to verify what the WAN router settings are. We will need to log in and check the settings. Perhaps we can start setting it up from scratch as an example. --- ## Set up your WAN router to work with CARP: - Before connecting the ISP router box to your OPNSense, it is recommended to reset it to factory default to set it up as a router (L3) to which we connect both OPNSense boxes to get a local IP address from the WAN router’s DHCP. With CARP, only one OPNSense will be active at a time. - Connect a computer to it to see what IP address you get from its DHCP. In my case, from Windows, I ran ‘ipconfig’ and found out that the gateway is 192.168.0.1:

- Log into the gateway’s web interface. Typically you get a default username and password that is written on the device or you can Google the provider name + device name to see what the defaults usually are.

- Remember to change the default password as soon as you manage to log in. - Typically you will have your own dedicated AP after the OPNSense firewall, so disable WiFi on the consumer modem/router.

- Set up a DHCP range that fits your network. In my case, my home LAN is on 192.168.8.0/24, so WAN1 LAN will be 192.168.80.0/24. The ISP device will likely reboot after that, so give it a while and test verify that your computer got a new IP address.

- Run ‘ipconfig /release’ and ‘ipconfig /renew’ if on a Windows station.

- Log back into it and let us set up static IP addresses for each of the MAC addresses of the adapters on your OPSense that will connect to it (each box runs an ethernet cable to the WAN1 ISP box). - Look at the MAC address used by each OPNSense box for the WAN1 interface:

- Then add it into the ISP device as reserved:

- Disable firewall capabilities on the ISP router/modem, since we will be handling it within OPNSense.

- For the same reason, enable UPnP:

- If you need to port forward other services such as for web hosting (ports 80 + 443), you will need to set it up in such a way that the external incoming port is then changed to a different internal incoming port to match that of your reverse proxy. - For instance, for an HAProxy service that is already running, it is possible to set it up to receive communication on port 8080 (HTTP) and 8443 (HTTPS). Apart from creating the required rules on both OPNSense boxes, when it comes to your WAN router, you will need to forward the ports to virtual IP interface of your OPNSense. Examples are below: - HTTP - external 80, internal 8080, forward to 192.168.80.254 (= WAN1 virtual IP) - HTTPS - external 443, internal 8443, forward to 192.168.80.254 (= WAN1 virtual IP) - SMTP (mail exchange) - external 25, internal 25, forward to 192.168.80.254 (= WAN1 virtual IP). - SMTPS - 465 - IMAP - 993 - …and anything else you need…

- Then remember to change the firewall rule on your OPNSense units, accordingly.

- And your HAProxy will need to receive communication NOT on port 443 (as that is used for XML-RPC to sync configs between OPNSense nodes) but modified to the port that you are forwarding from the WAN router(s). To modify that, go to 'Services' -> 'HAProxy' -> 'Settings' -> 'Virtual Services' tab -> 'Public Services'.

## Verify CARP functionality on LAN + WAN - Once you have verified that the virtual IP interfaces are set up on your OPNSense boxes for the LAN interfaces, you will be able to reach the current-main unit from a shared IP web interface. In this case, this is https://192.168.8.254. - Connect each OPNSense to the WAN1 router, which should result in each OPNSense unit getting an IP address from the DHCP server of the WAN1 router that we configured before based on its MAC addresses (no need to have static IP set on OPNSense for the WAN interfaces). - Open the virtual IP of your OPNSense (or log in directly to the web interface of your primary unit) and head to Interfaces → Overview and find out if your WAN interface got an IP address and the related CARP VHID group is linked to it.

- Let’s ensure that CARP for WAN1 has kicked in correctly. On each OPNSense unit, go to ‘Interfaces’ → Virtual IPs → Status. - On the main unit, CARP the status should be as ‘Master’:

- On the backup unit(s), CARP should indicate ‘Backup’:

### Troubleshooting CARP on WAN interface - If the above is different, then go to Virtual IPs → Settings and double check for this interface (i.e. WAN1) that they are: - In the same VHID for this interface on each OPNSense unit - The network address is the same on each (since they are sharing it) - The password matches - re-type it on each - All units have the same advbase number. - Switch to the ‘Advanced mode’ and set a higher number for ‘advskew’ on the backup units, leaving a low number on the main one (such as 50 on backup(s) versus 1 on the main). - Go to Virtual IPs → Status and click on the ‘Temporarily disable CARP’ and re-enable it again. Let’s see if it kicks in this time. - In case your CARP troubles are continuing, most likely, there is a firewall rule missing or misconfigured. On each OPNSense unit, go ‘Firewall’ → ‘Rules’ and check that it is configured according to the guide earlier. - Also, confirm that there are not NAT rules that could be interfering - Connect a computer directly to the WAN1 router. You will get a local IP on the WAN DHCP subnet such as 192.168.80.22/24. Try pinging each OPNSense device’s WAN1 interface.

- CARP output - from each OPSense unit’s shell, run ‘ifconfig | grep -A4 carp’ and compare the output. This is just another way besides the GUI to verify that the values match and that the advskew values are set up differently between main and backup(s).

---