High Availability & Two-Host Cluster Setup
Achieve 99.999% uptime with zero connection drops. Axonwall pairs industry-standard Keepalived VRRPv2/v3 clustering with conntrackd FTFW kernel state replication, mutual-TLS signed configuration & SQLite WAL replication, Chrony NTP peering, OpenLDAP multi-provider syncrepl, incremental OpenZFS snapshot streaming, and automated 2-node K3s joining.
Cluster Architecture & Full-Stack Synchronization
An Axonwall HA cluster consists of two appliances (master and backup) connected to shared upstream WAN and downstream LAN broadcast domains, plus a dedicated point-to-point synchronization link (recommended on eth3). Beyond floating VRRP virtual IPs and kernel TCP/UDP connection tracking, Axonwall synchronizes the entire appliance stack across the dedicated HA link:
+---------------------------------------------------------------------------------+
| Upstream WAN Switch / ISP |
| Floating WAN VIP: 198.51.100.1/24 (VRID 51) |
+---------------------------------------------------------------------------------+
| |
| eth0: 198.51.100.2/24 | eth0: 198.51.100.3/24
v v
+------------------------------+ +------------------------------+
| Host 1: axonwall-fw01 | | Host 2: axonwall-fw02 |
| Role: MASTER (Priority 150) | | Role: BACKUP (Priority 100) |
| ZFS Pool: Read-Write | | ZFS Pool: Read-Only Target |
| Headscale: Active | | Headscale: Standby (Stopped)|
+------------------------------+ +------------------------------+
^ ^
| Dedicated HA Sync Link (eth3) |
| 10.0.99.1/30 <=====> 10.0.99.2/30 |
+<=====================================>+
| • VRRP Heartbeats (IP Proto 112) |
| • conntrackd FTFW State (UDP :3780) |
| • HA Sync & SQLite WAL (mTLS :9876) |
| • Chrony Symmetric NTP (UDP :123) |
| • OpenLDAP syncrepl (LDAPS :636) |
| • Kea DHCPv4 Hot-Standby (TCP :8001) |
v v
+------------------------------+ +------------------------------+
| eth1: 10.0.10.2/24 | eth1: 10.0.10.3/24
v v
+---------------------------------------------------------------------------------+
| LAN Core Switch |
| Floating LAN Gateway VIP: 10.0.10.1/24 (VRID 52) |
+---------------------------------------------------------------------------------+| Subsystem | Port / Protocol | Direction & Mechanism | Failover Action |
|---|---|---|---|
| Keepalived VRRP | IP Proto 112 | VRRP sync group (VG1) tracks WAN + LAN VIPs together | Promotes VIPs in <1s and invokes axonwall-ha-notify via /run/axonwall/ha.sock |
| conntrackd FTFW | UDP 3780 | Sequence-numbered Netfilter connection tracking delta stream | conntrackd -c commits external cache into kernel table |
| Axonwall HA Engine | TCP 9876 (mTLS) | Signed desired-state (/api/v1/sync/state), SQLite (/api/v1/sync/db) & PKI (/api/v1/sync/pki-enroll) | Atomically promotes state.json.ha-replica & verified SQLite DBs, then starts headscale |
| Chrony NTP Peer | UDP 123 | Symmetric peer <peerIp> iburst time discipline | Prevents OpenLDAP CSN & SQLite timestamp drift during WAN outage |
| OpenLDAP Directory | TCP 636 (LDAPS) | refreshAndPersist multi-provider syncrepl (ServerID 1 & 2) | Seamless local LDAP/802.1X/VPN authentication on both nodes |
| Kea DHCPv4 HA | TCP 8001 | libdhcp_ha.so hot-standby lease replication | Zero duplicate IP assignments when Primary goes offline |
| OpenZFS Storage | TCP 9876 (mTLS) | Incremental zfs send -w -I snapshot stream (/api/v1/sync/zfs) | Flips target pool readonly=off and starts SMB/NFS/iSCSI |
Step-by-Step Guide: Setting Up Two Hosts
Follow this 8-step guide to deploy two Axonwall appliances—axonwall-fw01 (Primary MASTER) and axonwall-fw02 (Secondary BACKUP)—into a zero-drop active/passive cluster.
Plan Physical Cabling, Interface Names & IP Addressing
Both hosts must have identical Linux network device mappings (eth0 for WAN, eth1 for LAN, and eth3 for the dedicated HA sync interconnect) so that synchronized nftables rulesets and VRRP definitions reference the exact same logical and physical interface names on both machines.
| Logical / Device (Role) | Host 1 (axonwall-fw01) | Host 2 (axonwall-fw02) | Floating VRRP VIP (Shared) |
|---|---|---|---|
| wan / eth0 (WAN Uplink) | 198.51.100.2/24 | 198.51.100.3/24 | 198.51.100.1/24 (VRID 51) |
| lan / eth1 (LAN Core) | 10.0.10.2/24 | 10.0.10.3/24 | 10.0.10.1/24 (VRID 52) |
| ha-sync / eth3 (Dedicated HA Sync) | 10.0.99.1/30 | 10.0.99.2/30 | None (Point-to-Point Direct) |
eth0 and eth1 vNICs and disable any hypervisor firewall rules that drop IP Protocol 112 (VRRP) or multicast 224.0.0.18 / 225.0.0.50. Connect eth3 to an isolated host-only bridge or dedicated physical NIC port.Configure Physical & Sync Interfaces on Both Hosts
Each host retains its own physical interface IP addresses in /etc/axonwall/interfaces.yaml. By design, syncSections.interfaces defaults to false so Host 1 never overwrites Host 2's unique physical management IPs during state synchronization. Note that each interface entry requires both a logical name and a kernel device:
interfaces:
- name: wan
device: eth0
role: wan
type: static
enabled: true
ipv4Address: 198.51.100.2/24
ipv4Gateway: 198.51.100.254
mtu: 1500
description: Primary WAN Uplink (Host 1)
- name: lan
device: eth1
role: lan
type: static
enabled: true
ipv4Address: 10.0.10.2/24
mtu: 1500
description: Core LAN Interface (Host 1)
- name: ha-sync
device: eth3
role: ha-sync
type: static
enabled: true
ipv4Address: 10.0.99.1/30
mtu: 9000
description: Dedicated HA Sync Link (to fw02)interfaces:
- name: wan
device: eth0
role: wan
type: static
enabled: true
ipv4Address: 198.51.100.3/24
ipv4Gateway: 198.51.100.254
mtu: 1500
description: Secondary WAN Uplink (Host 2)
- name: lan
device: eth1
role: lan
type: static
enabled: true
ipv4Address: 10.0.10.3/24
mtu: 1500
description: Core LAN Interface (Host 2)
- name: ha-sync
device: eth3
role: ha-sync
type: static
enabled: true
ipv4Address: 10.0.99.2/30
mtu: 9000
description: Dedicated HA Sync Link (to fw01)Verify point-to-point reachability across eth3 before enabling HA: run ping -I eth3 -c 3 10.0.99.2 from Host 1 and ping -I eth3 -c 3 10.0.99.1 from Host 2.
Configure Host 1 (axonwall-fw01) as Primary MASTER
Generate a shared secret on Host 1 (openssl rand -hex 24) for the initial PKI bootstrap and VRRP authentication, then populate /etc/axonwall/ha.yaml under the top-level ha: key on axonwall-fw01 with nodeRole: master and priority 150:
ha:
enabled: true
nodeRole: master
role: master
syncInterface: eth3
peerAddress: 10.0.99.2
peerIp: 10.0.99.2
peerPort: 2966
maintenanceMode: false
vrrpInstances:
- name: VI_WAN
vrid: 51
interface: eth0
priority: 150
virtualIps:
- 198.51.100.1/24
authPassword: "replace-with-shared-secret"
- name: VI_LAN
vrid: 52
interface: eth1
priority: 150
virtualIps:
- 10.0.10.1/24
authPassword: "replace-with-shared-secret"
conntrackSync:
enabled: true
interface: eth3
mode: FTFW
protocol: multicast
multicastGroup: 225.0.0.50
peerAddress: 10.0.99.2
port: 3780
sqliteSync:
enabled: true
mode: direct_stream
syncIntervalSeconds: 5
listenPort: 9876
peerAddress: 10.0.99.2
peerPort: 9876
authToken: "replace-with-shared-secret"
transport: mtls
requireSignedPayloads: true
databases:
- /var/lib/axonwall/headscale/db.sqlite
syncSections:
interfaces: false
firewall: true
dhcp: true
dns: true
aliases: true
vpn: true
sqlite: true
nas: true
k3s: trueConfigure Host 2 (axonwall-fw02) as Secondary BACKUP
On axonwall-fw02, configure /etc/axonwall/ha.yaml with nodeRole: backup, a lower VRRP priority (100), peerAddress: 10.0.99.1 pointing back to Host 1, and the exact same authPassword / authToken and vrrpInstances:
ha:
enabled: true
nodeRole: backup
role: backup
syncInterface: eth3
peerAddress: 10.0.99.1
peerIp: 10.0.99.1
peerPort: 2966
maintenanceMode: false
vrrpInstances:
- name: VI_WAN
vrid: 51
interface: eth0
priority: 100
virtualIps:
- 198.51.100.1/24
authPassword: "replace-with-shared-secret"
- name: VI_LAN
vrid: 52
interface: eth1
priority: 100
virtualIps:
- 10.0.10.1/24
authPassword: "replace-with-shared-secret"
conntrackSync:
enabled: true
interface: eth3
mode: FTFW
protocol: multicast
multicastGroup: 225.0.0.50
peerAddress: 10.0.99.1
port: 3780
sqliteSync:
enabled: true
mode: direct_stream
syncIntervalSeconds: 5
listenPort: 9876
peerAddress: 10.0.99.1
peerPort: 9876
authToken: "replace-with-shared-secret"
transport: mtls
requireSignedPayloads: true
databases:
- /var/lib/axonwall/headscale/db.sqlite
syncSections:
interfaces: false
firewall: true
dhcp: true
dns: true
aliases: true
vpn: true
sqlite: true
nas: true
k3s: trueApply Configuration via Web Console, CLI, or Neovim Super Configurator
You can configure and commit the HA configuration on both hosts using any of Axonwall's three management interfaces—they all read and write the exact same /etc/axonwall/ha.yaml state and trigger the privileged axonwalld compiler:
Navigate to /ha on Host 1 and Host 2, toggle Failover Configuration, enter the Node Role, VRID, Sync Interface (eth3), Peer Sync IP, Sync Password, Virtual IP (CIDR), and VRRP Interface (eth1), enable Distributed SQLite & Mesh State Sync, and click Save to ha.yaml.
Over SSH or serial console, launch the form-driven TUI directly onto the High Availability domain:
axonwallctl config haPress Ctrl+s to validate & apply with 120s Commit-Confirm safety.
Validate and apply /etc/axonwall/*.yaml on Host 1 first, then Host 2:
axonwallctl validate && axonwallctl apply --no-confirm && axonwallctl confirmaxonwalld applies ha.yaml, it automatically compiles and reloads:/etc/keepalived/keepalived.conf+/usr/local/bin/axonwall-ha-notifytransition hook (delivering state transitions over/run/axonwall/ha.sock)/etc/conntrackd/conntrackd.conf(FTFW state replication overeth3)/etc/chrony/conf.d/axonwall-ha.conf(symmetric NTP peering overeth3)- nftables rules permitting VRRP (
ip protocol 112), conntrackd (udp dport 3780), HA Sync (tcp dport 9876), Chrony NTP (udp dport 123), and OpenLDAPsyncrepl(tcp dport 636) strictly oniifname "eth3"
Verify Automatic HA PKI Enrollment, VIPs, conntrackd & State Sync
Within 5–15 seconds of enabling HA on both hosts, Host 2 (backup) polls Host 1 (master) on 10.0.99.1:9876 using the shared authToken, submits a CSR to /api/v1/sync/pki-enroll to enroll in the cluster ECDSA P-256 Root CA (/var/lib/axonwall/secrets/ha-pki/), and upgrades the replication transport to mutual TLS (mtls) with ECDSA-signed payload verification. Run the following checks on both hosts to confirm healthy cluster convergence:
# 1. Verify live HA Sync Trust state, effective transport, and SHA-256 PKI fingerprints: sudo curl -s --unix-socket /run/axonwall/agent-api.sock \ -H "X-Axonwall-Agent-Token: $(sudo cat /etc/axonwall/agent.token)" \ http://localhost/v1/ha/sync-trust | jq . # 2. Confirm Host 1 (MASTER) holds the Floating WAN & LAN VIPs (198.51.100.1 & 10.0.10.1): ip -br addr show dev eth0 ip -br addr show dev eth1 # 3. Verify Keepalived VRRP sync group (VG1) and conntrackd FTFW state cache: sudo systemctl status keepalived conntrackd sudo conntrackd -s # 4. Verify symmetric Chrony NTP peering across eth3: chronyc sources -v # 5. Query the Master HA Sync Server status endpoint directly over mTLS (port 9876): sudo curl -sk --cacert /var/lib/axonwall/secrets/ha-pki/ca.crt \ --cert /var/lib/axonwall/secrets/ha-pki/node.crt \ --key /var/lib/axonwall/secrets/ha-pki/node.key \ -H "X-Axonwall-Sync-Token: replace-with-shared-secret" \ https://10.0.99.1:9876/api/v1/sync/status | jq .
In the Web Console under /ha, check the Sync Trust (live from this node's daemon) panel inside the SQLite sync card: once PKI enrollment completes, PKI enrollment: enrolled and Transport in effect: mtls will appear, and both nodes will display identical Node certificate SHA-256 and HA CA SHA-256 fingerprints.
(Optional) Enable ZFS Snapshot Replication & 2-Node K3s + OpenEBS HA
If your two hosts also run the optional OpenZFS Storage (/etc/axonwall/storage.yaml) or K3s Kubernetes (/etc/axonwall/k3s.yaml) workloads, enable their HA peer integration so storage pools and container volumes fail over alongside the firewall:
nas:
enabled: true
pools:
- name: tank
topology: mirror
dataDisks:
- /dev/nvme0n1
- /dev/nvme1n1
compression: zstd
datasets:
- name: shares
pool: tank
type: filesystem
compression: zstd
quotaGb: 2048
replicateToHaPeer: true
snapshotPolicies:
- dataset: tank/shares
interval: hourly
keepCount: 24
replicateToHaPeer: true
enabled: truek3s:
enabled: true
clusterCidr: 10.42.0.0/16
serviceCidr: 10.43.0.0/16
nodeIpInterface: lan
flannelBackend: wireguard-native
autoJoinHaPeer: true
openEbs:
enabled: true
zfsLocalPvEnabled: true
zfsLocalPvPool: tank
replicatedStorageEnabled: true
defaultReplicaCount: 2
replicatedBackingPool: tank/openebs-replicated# Rewrite ~/.kube/config server endpoint to the floating LAN VRRP VIP (e.g. https://10.0.10.1:6443) # so kubectl and helm survive control-plane node failover without reconnecting: axonwall-kubeconfig --vip # Or via axonwallctl: axonwallctl kubeconfig --vip
Test Live Failover & Graceful Maintenance Mode
Before placing the cluster into production, validate both graceful maintenance demotion and abrupt hardware failover while keeping a continuous SSH session and ping running from a LAN workstation through the 10.0.10.1 floating gateway VIP:
# From a LAN client routed through VIP 10.0.10.1, start an active SSH & ping test: ping -i 0.2 1.1.1.1 # TEST A — Graceful Maintenance / Planned Failover (on Host 1 / axonwall-fw01): # Set "maintenanceMode: true" under ha: in /etc/axonwall/ha.yaml and run "axonwallctl apply" # (or click "Force Failover to Peer" -> "Save to ha.yaml" in the /ha Web Console). # Keepalived lowers effective priority and shifts VIPs + conntrack state to Host 2 with 0 packet loss. # Verify Host 2 (axonwall-fw02) now owns 198.51.100.1 and 10.0.10.1: ssh admin@10.0.10.3 "ip -br addr show dev eth0; ip -br addr show dev eth1" # TEST B — Unplanned Link / Power Loss Simulation (on Host 1 / axonwall-fw01): # Bring down the WAN link or stop keepalived on Host 1: sudo ip link set eth0 down # Host 2 transitions to MASTER within 1000ms, commits conntrackd cache, and active TCP/SSH stays up. # Restore Host 1 WAN link: sudo ip link set eth0 up
Cluster PKI, Mutual TLS & Signed State Payloads
Axonwall never transmits configuration state or SQLite WAL frames in cleartext once enrolled. The Go daemon (axonwalld) maintains a dedicated ECDSA P-256 High Availability PKI hierarchy stored in /var/lib/axonwall/secrets/ha-pki/:
Automated CSR Enrollment & Shared Identity
The Primary node generates a 10-year ECDSA P-256 Root CA (ca.crt) and node certificate (node.crt) bound to the cluster's sync IP Subject Alternative Names (SANs). On its first poll, the Backup node submits a CSR over the token-authenticated bootstrap channel to /api/v1/sync/pki-enroll, installs the signed bundle and OpenLDAP syncrepl credentials, and switches to mutual TLS (mtls).
Ed25519 / ECDSA Payload Signing & Safe Cert Rotation
Every /api/v1/sync/state, DHCP lease bundle, and /api/v1/sync/db response on :9876 carries both an X-Checksum-SHA256 digest and a detached Ed25519/ECDSA X-Axonwall-Payload-Signature header. When requireSignedPayloads: true is enabled, unsigned or tampered payloads are strictly rejected before touching disk. Administrators can rotate leaf hub/sync certificates without invalidating Operator-CA trust anchors using sudo axonwall-rotate-hub-cert.
Keepalived VRRP v2/v3 Clustering
Keepalived manages virtual IP address failover according to RFC 5798. Axonwall groups all configured VRRP instances into a unified vrrp_sync_group VG1 so that if either the WAN uplink (eth0) or LAN trunk (eth1) loses carrier, all virtual IPs fail over together—preventing asymmetric blackhole routing.
VRRP Failover Triggers
- Loss of physical link carrier on WAN or LAN VRRP interfaces
- Unresponsive
axonwalldhealth probe (chk_axonwalld) - Manual maintenance failover (
maintenanceMode: trueor Role switch in/ha) - Power loss or kernel watchdog reset on the active Master
Transition Orchestration Hook
On state change, Keepalived runs conntrackd -c/-f/-t and invokes /usr/local/bin/axonwall-ha-notify, which notifies axonwalld over the root-authenticated Unix socket /run/axonwall/ha.sock to promote verified state replicas, flip OpenZFS pool readonly flags, and start or stop headscale to prevent dual-active split-brain.
conntrackd Stateful Replication Engine
Without state synchronization, a VRRP failover forces all active TCP connections to reset, cutting off active SSH sessions, dropping VoIP calls, and aborting file downloads. Axonwall integrates conntrackd to replicate the kernel Netfilter state table seamlessly across eth3.
FTFW Protocol (Fault-Tolerant Framework)
Unlike BSD pfsync which uses simple unacknowledged UDP broadcasts, the Netfilter FTFW protocol implements sequence-numbered state transactions, explicit sequence acknowledgment, and reliable retransmission of lost state deltas over either multicast (225.0.0.50:3780) or point-to-point unicast UDP.
Sync {
Mode FTFW {
DisableExternalCache Off
CommitTimeout 1800
PurgeTimeout 5
}
Multicast {
IPv4_address 225.0.0.50
IPv4_interface 10.0.99.1
Interface eth3
SndSocketBuffer 1249280
RcvSocketBuffer 1249280
Checksum on
}
}
General {
HashSize 32768
HashLimit 131072
LogFile on
Syslog on
LockFile /var/lock/conntrack.lock
UNIX {
Path /var/run/conntrackd.ctl
Backlog 20
}
NetlinkBufferSize 2097152
NetlinkBufferSizeMaxGrowth 8388608
Filter From Userspace {
Protocol Accept {
TCP
UDP
ICMP
}
Address Ignore {
IPv4_address 127.0.0.1
}
}
}SQLite WAL, Chrony NTP, OpenLDAP, ZFS & K3s Replication
Axonwall extends High Availability beyond L3/L4 packet filtering so stateful control-plane services survive node failure without data loss or split-brain corruption:
SQLite Hot VACUUM INTO & State Replica Promotion
The Primary node creates consistent lock-free hot snapshots via VACUUM INTO (and WAL checkpointing) of allowed databases under /var/lib/axonwall/ (such as /var/lib/axonwall/headscale/db.sqlite) and streams them with SHA-256 and Ed25519/ECDSA signature verification every syncIntervalSeconds seconds. Before replacing a live file, the Backup node validates PRAGMA integrity_check, preserves service UID/GID ownership, and stages state.json.ha-replica for atomic promotion upon MASTER takeover.
Chrony Symmetric NTP & OpenLDAP N-Way syncrepl
OpenLDAP N-Way Multi-Provider syncrepl over ldaps://:636 relies on Change Sequence Numbers (CSNs) with microsecond timestamps. Axonwall automatically writes /etc/chrony/conf.d/axonwall-ha.conf with symmetric peering (peer <peerIp> iburst) on eth3 so both nodes maintain sub-millisecond clock alignment even if upstream internet NTP servers are unreachable.
Incremental OpenZFS Send/Recv
Datasets and snapshot policies marked with replicateToHaPeer: true in /etc/axonwall/storage.yaml are snapshotted (@ha-sync-<ns>) and streamed via zfs send -w -I | zfs recv over /api/v1/sync/zfs to the Backup node. On the Backup node, replicated pools stay readonly=on until Keepalived promotes the node to MASTER.
Embedded-etcd K3s HA Join & Mayastor Volumes
When autoJoinHaPeer: true is set in /etc/axonwall/k3s.yaml, the Backup node securely fetches the K3s embedded-etcd join token from /api/v1/sync/k3s-token over mTLS, joins the cluster over the WireGuard Flannel mesh with axonwall-replicated-ha OpenEBS Mayastor NVMe-oF volumes, and exposes an HA-resilient kubeconfig via axonwall-kubeconfig --vip (axonwallctl kubeconfig --vip).
Failover Latency & Connection Continuity
During an abrupt power failure on the Master appliance, the Backup node detects the missed VRRP advertisement within 1000ms, transitions to MASTER, and flushes the external state cache directly into the Linux Netfilter kernel table:
| Protocol / Session Type | Behavior on Legacy BSD Firewalls | Behavior on Axonwall HA |
|---|---|---|
| Active TCP (SSH, HTTPS, Database) | Broken on packet drop / retransmission limit | Zero-drop; seamlessly continues without RST |
| VoIP Calls (SIP Signaling / RTP Audio) | Audio drops; SIP session terminates | Sub-second jitter; call remains connected |
| WireGuard Site-to-Site Tunnels | Requires 30s re-handshake | Instantaneous handover; zero packet drop |
net.netfilter.nf_conntrack_tcp_timeout_established = 86400 and expands nf_conntrack_max = 2097152 to handle enterprise connection tables up to 2 million concurrent flows without memory thrashing.