Clustering, 6-Plane State Replication & Two-Host Walkthrough • v0.13.0

High Availability & Two-Host Cluster Setup

Achieve 99.999% uptime with zero connection drops. Axonwall pairs industry-standard Keepalived VRRPv2/v3 clustering with conntrackd FTFW kernel state replication, mutual-TLS signed configuration & SQLite WAL replication, Chrony NTP peering, OpenLDAP multi-provider syncrepl, incremental OpenZFS snapshot streaming, and automated 2-node K3s joining.

Cluster Architecture & Full-Stack Synchronization

An Axonwall HA cluster consists of two appliances (master and backup) connected to shared upstream WAN and downstream LAN broadcast domains, plus a dedicated point-to-point synchronization link (recommended on eth3). Beyond floating VRRP virtual IPs and kernel TCP/UDP connection tracking, Axonwall synchronizes the entire appliance stack across the dedicated HA link:

High Availability Two-Host Full-Stack Topology
text
+---------------------------------------------------------------------------------+
|                              Upstream WAN Switch / ISP                          |
|                         Floating WAN VIP: 198.51.100.1/24 (VRID 51)             |
+---------------------------------------------------------------------------------+
                      |                                       |
                      | eth0: 198.51.100.2/24                 | eth0: 198.51.100.3/24
                      v                                       v
       +------------------------------+        +------------------------------+
       |  Host 1: axonwall-fw01       |        |  Host 2: axonwall-fw02       |
       |  Role: MASTER (Priority 150) |        |  Role: BACKUP (Priority 100) |
       |  ZFS Pool: Read-Write        |        |  ZFS Pool: Read-Only Target  |
       |  Headscale: Active           |        |  Headscale: Standby (Stopped)|
       +------------------------------+        +------------------------------+
                      ^                                       ^
                      |      Dedicated HA Sync Link (eth3)    |
                      |   10.0.99.1/30 <=====> 10.0.99.2/30   |
                      +<=====================================>+
                      |  • VRRP Heartbeats (IP Proto 112)     |
                      |  • conntrackd FTFW State (UDP :3780)  |
                      |  • HA Sync & SQLite WAL (mTLS :9876)  |
                      |  • Chrony Symmetric NTP (UDP :123)    |
                      |  • OpenLDAP syncrepl (LDAPS :636)     |
                      |  • Kea DHCPv4 Hot-Standby (TCP :8001) |
                      v                                       v
       +------------------------------+        +------------------------------+
                      | eth1: 10.0.10.2/24                    | eth1: 10.0.10.3/24
                      v                                       v
+---------------------------------------------------------------------------------+
|                                 LAN Core Switch                                 |
|                Floating LAN Gateway VIP: 10.0.10.1/24 (VRID 52)                 |
+---------------------------------------------------------------------------------+
SubsystemPort / ProtocolDirection & MechanismFailover Action
Keepalived VRRPIP Proto 112VRRP sync group (VG1) tracks WAN + LAN VIPs togetherPromotes VIPs in <1s and invokes axonwall-ha-notify via /run/axonwall/ha.sock
conntrackd FTFWUDP 3780Sequence-numbered Netfilter connection tracking delta streamconntrackd -c commits external cache into kernel table
Axonwall HA EngineTCP 9876 (mTLS)Signed desired-state (/api/v1/sync/state), SQLite (/api/v1/sync/db) & PKI (/api/v1/sync/pki-enroll)Atomically promotes state.json.ha-replica & verified SQLite DBs, then starts headscale
Chrony NTP PeerUDP 123Symmetric peer <peerIp> iburst time disciplinePrevents OpenLDAP CSN & SQLite timestamp drift during WAN outage
OpenLDAP DirectoryTCP 636 (LDAPS)refreshAndPersist multi-provider syncrepl (ServerID 1 & 2)Seamless local LDAP/802.1X/VPN authentication on both nodes
Kea DHCPv4 HATCP 8001libdhcp_ha.so hot-standby lease replicationZero duplicate IP assignments when Primary goes offline
OpenZFS StorageTCP 9876 (mTLS)Incremental zfs send -w -I snapshot stream (/api/v1/sync/zfs)Flips target pool readonly=off and starts SMB/NFS/iSCSI
STEP-BY-STEP WALKTHROUGH

Step-by-Step Guide: Setting Up Two Hosts

Follow this 8-step guide to deploy two Axonwall appliances—axonwall-fw01 (Primary MASTER) and axonwall-fw02 (Secondary BACKUP)—into a zero-drop active/passive cluster.

1

Plan Physical Cabling, Interface Names & IP Addressing

Both hosts must have identical Linux network device mappings (eth0 for WAN, eth1 for LAN, and eth3 for the dedicated HA sync interconnect) so that synchronized nftables rulesets and VRRP definitions reference the exact same logical and physical interface names on both machines.

Logical / Device (Role)Host 1 (axonwall-fw01)Host 2 (axonwall-fw02)Floating VRRP VIP (Shared)
wan / eth0 (WAN Uplink)198.51.100.2/24198.51.100.3/24198.51.100.1/24 (VRID 51)
lan / eth1 (LAN Core)10.0.10.2/2410.0.10.3/2410.0.10.1/24 (VRID 52)
ha-sync / eth3 (Dedicated HA Sync)10.0.99.1/3010.0.99.2/30None (Point-to-Point Direct)
Virtualization Note (Proxmox VE, KVM/libvirt, VMware ESXi)
If either host runs inside a hypervisor, enable MAC Spoofing / Promiscuous Mode / Forged Transmits on the eth0 and eth1 vNICs and disable any hypervisor firewall rules that drop IP Protocol 112 (VRRP) or multicast 224.0.0.18 / 225.0.0.50. Connect eth3 to an isolated host-only bridge or dedicated physical NIC port.
2

Configure Physical & Sync Interfaces on Both Hosts

Each host retains its own physical interface IP addresses in /etc/axonwall/interfaces.yaml. By design, syncSections.interfaces defaults to false so Host 1 never overwrites Host 2's unique physical management IPs during state synchronization. Note that each interface entry requires both a logical name and a kernel device:

Host 1 (axonwall-fw01): /etc/axonwall/interfaces.yaml
yaml
interfaces:
  - name: wan
    device: eth0
    role: wan
    type: static
    enabled: true
    ipv4Address: 198.51.100.2/24
    ipv4Gateway: 198.51.100.254
    mtu: 1500
    description: Primary WAN Uplink (Host 1)

  - name: lan
    device: eth1
    role: lan
    type: static
    enabled: true
    ipv4Address: 10.0.10.2/24
    mtu: 1500
    description: Core LAN Interface (Host 1)

  - name: ha-sync
    device: eth3
    role: ha-sync
    type: static
    enabled: true
    ipv4Address: 10.0.99.1/30
    mtu: 9000
    description: Dedicated HA Sync Link (to fw02)
Host 2 (axonwall-fw02): /etc/axonwall/interfaces.yaml
yaml
interfaces:
  - name: wan
    device: eth0
    role: wan
    type: static
    enabled: true
    ipv4Address: 198.51.100.3/24
    ipv4Gateway: 198.51.100.254
    mtu: 1500
    description: Secondary WAN Uplink (Host 2)

  - name: lan
    device: eth1
    role: lan
    type: static
    enabled: true
    ipv4Address: 10.0.10.3/24
    mtu: 1500
    description: Core LAN Interface (Host 2)

  - name: ha-sync
    device: eth3
    role: ha-sync
    type: static
    enabled: true
    ipv4Address: 10.0.99.2/30
    mtu: 9000
    description: Dedicated HA Sync Link (to fw01)

Verify point-to-point reachability across eth3 before enabling HA: run ping -I eth3 -c 3 10.0.99.2 from Host 1 and ping -I eth3 -c 3 10.0.99.1 from Host 2.

3

Configure Host 1 (axonwall-fw01) as Primary MASTER

Generate a shared secret on Host 1 (openssl rand -hex 24) for the initial PKI bootstrap and VRRP authentication, then populate /etc/axonwall/ha.yaml under the top-level ha: key on axonwall-fw01 with nodeRole: master and priority 150:

Host 1 (axonwall-fw01): /etc/axonwall/ha.yaml
yaml
ha:
  enabled: true
  nodeRole: master
  role: master
  syncInterface: eth3
  peerAddress: 10.0.99.2
  peerIp: 10.0.99.2
  peerPort: 2966
  maintenanceMode: false

  vrrpInstances:
    - name: VI_WAN
      vrid: 51
      interface: eth0
      priority: 150
      virtualIps:
        - 198.51.100.1/24
      authPassword: "replace-with-shared-secret"
    - name: VI_LAN
      vrid: 52
      interface: eth1
      priority: 150
      virtualIps:
        - 10.0.10.1/24
      authPassword: "replace-with-shared-secret"

  conntrackSync:
    enabled: true
    interface: eth3
    mode: FTFW
    protocol: multicast
    multicastGroup: 225.0.0.50
    peerAddress: 10.0.99.2
    port: 3780

  sqliteSync:
    enabled: true
    mode: direct_stream
    syncIntervalSeconds: 5
    listenPort: 9876
    peerAddress: 10.0.99.2
    peerPort: 9876
    authToken: "replace-with-shared-secret"
    transport: mtls
    requireSignedPayloads: true
    databases:
      - /var/lib/axonwall/headscale/db.sqlite

  syncSections:
    interfaces: false
    firewall: true
    dhcp: true
    dns: true
    aliases: true
    vpn: true
    sqlite: true
    nas: true
    k3s: true
4

Configure Host 2 (axonwall-fw02) as Secondary BACKUP

On axonwall-fw02, configure /etc/axonwall/ha.yaml with nodeRole: backup, a lower VRRP priority (100), peerAddress: 10.0.99.1 pointing back to Host 1, and the exact same authPassword / authToken and vrrpInstances:

Host 2 (axonwall-fw02): /etc/axonwall/ha.yaml
yaml
ha:
  enabled: true
  nodeRole: backup
  role: backup
  syncInterface: eth3
  peerAddress: 10.0.99.1
  peerIp: 10.0.99.1
  peerPort: 2966
  maintenanceMode: false

  vrrpInstances:
    - name: VI_WAN
      vrid: 51
      interface: eth0
      priority: 100
      virtualIps:
        - 198.51.100.1/24
      authPassword: "replace-with-shared-secret"
    - name: VI_LAN
      vrid: 52
      interface: eth1
      priority: 100
      virtualIps:
        - 10.0.10.1/24
      authPassword: "replace-with-shared-secret"

  conntrackSync:
    enabled: true
    interface: eth3
    mode: FTFW
    protocol: multicast
    multicastGroup: 225.0.0.50
    peerAddress: 10.0.99.1
    port: 3780

  sqliteSync:
    enabled: true
    mode: direct_stream
    syncIntervalSeconds: 5
    listenPort: 9876
    peerAddress: 10.0.99.1
    peerPort: 9876
    authToken: "replace-with-shared-secret"
    transport: mtls
    requireSignedPayloads: true
    databases:
      - /var/lib/axonwall/headscale/db.sqlite

  syncSections:
    interfaces: false
    firewall: true
    dhcp: true
    dns: true
    aliases: true
    vpn: true
    sqlite: true
    nas: true
    k3s: true
5

Apply Configuration via Web Console, CLI, or Neovim Super Configurator

You can configure and commit the HA configuration on both hosts using any of Axonwall's three management interfaces—they all read and write the exact same /etc/axonwall/ha.yaml state and trigger the privileged axonwalld compiler:

1. Web Console (/ha)

Navigate to /ha on Host 1 and Host 2, toggle Failover Configuration, enter the Node Role, VRID, Sync Interface (eth3), Peer Sync IP, Sync Password, Virtual IP (CIDR), and VRRP Interface (eth1), enable Distributed SQLite & Mesh State Sync, and click Save to ha.yaml.

2. Neovim Super Configurator

Over SSH or serial console, launch the form-driven TUI directly onto the High Availability domain:

axonwallctl config ha

Press Ctrl+s to validate & apply with 120s Commit-Confirm safety.

3. axonwallctl CLI

Validate and apply /etc/axonwall/*.yaml on Host 1 first, then Host 2:

axonwallctl validate && axonwallctl apply --no-confirm && axonwallctl confirm
What Happens Automatically When You Apply HA
When axonwalld applies ha.yaml, it automatically compiles and reloads:
  • /etc/keepalived/keepalived.conf + /usr/local/bin/axonwall-ha-notify transition hook (delivering state transitions over /run/axonwall/ha.sock)
  • /etc/conntrackd/conntrackd.conf (FTFW state replication over eth3)
  • /etc/chrony/conf.d/axonwall-ha.conf (symmetric NTP peering over eth3)
  • nftables rules permitting VRRP (ip protocol 112), conntrackd (udp dport 3780), HA Sync (tcp dport 9876), Chrony NTP (udp dport 123), and OpenLDAP syncrepl (tcp dport 636) strictly on iifname "eth3"
6

Verify Automatic HA PKI Enrollment, VIPs, conntrackd & State Sync

Within 5–15 seconds of enabling HA on both hosts, Host 2 (backup) polls Host 1 (master) on 10.0.99.1:9876 using the shared authToken, submits a CSR to /api/v1/sync/pki-enroll to enroll in the cluster ECDSA P-256 Root CA (/var/lib/axonwall/secrets/ha-pki/), and upgrades the replication transport to mutual TLS (mtls) with ECDSA-signed payload verification. Run the following checks on both hosts to confirm healthy cluster convergence:

Cluster Verification Checklist (Run on Host 1 & Host 2)
bash
# 1. Verify live HA Sync Trust state, effective transport, and SHA-256 PKI fingerprints:
sudo curl -s --unix-socket /run/axonwall/agent-api.sock \
  -H "X-Axonwall-Agent-Token: $(sudo cat /etc/axonwall/agent.token)" \
  http://localhost/v1/ha/sync-trust | jq .

# 2. Confirm Host 1 (MASTER) holds the Floating WAN & LAN VIPs (198.51.100.1 & 10.0.10.1):
ip -br addr show dev eth0
ip -br addr show dev eth1

# 3. Verify Keepalived VRRP sync group (VG1) and conntrackd FTFW state cache:
sudo systemctl status keepalived conntrackd
sudo conntrackd -s

# 4. Verify symmetric Chrony NTP peering across eth3:
chronyc sources -v

# 5. Query the Master HA Sync Server status endpoint directly over mTLS (port 9876):
sudo curl -sk --cacert /var/lib/axonwall/secrets/ha-pki/ca.crt \
  --cert /var/lib/axonwall/secrets/ha-pki/node.crt \
  --key /var/lib/axonwall/secrets/ha-pki/node.key \
  -H "X-Axonwall-Sync-Token: replace-with-shared-secret" \
  https://10.0.99.1:9876/api/v1/sync/status | jq .

In the Web Console under /ha, check the Sync Trust (live from this node's daemon) panel inside the SQLite sync card: once PKI enrollment completes, PKI enrollment: enrolled and Transport in effect: mtls will appear, and both nodes will display identical Node certificate SHA-256 and HA CA SHA-256 fingerprints.

7

(Optional) Enable ZFS Snapshot Replication & 2-Node K3s + OpenEBS HA

If your two hosts also run the optional OpenZFS Storage (/etc/axonwall/storage.yaml) or K3s Kubernetes (/etc/axonwall/k3s.yaml) workloads, enable their HA peer integration so storage pools and container volumes fail over alongside the firewall:

ZFS HA Replication (/etc/axonwall/storage.yaml)
yaml
nas:
  enabled: true
  pools:
    - name: tank
      topology: mirror
      dataDisks:
        - /dev/nvme0n1
        - /dev/nvme1n1
      compression: zstd
  datasets:
    - name: shares
      pool: tank
      type: filesystem
      compression: zstd
      quotaGb: 2048
      replicateToHaPeer: true
  snapshotPolicies:
    - dataset: tank/shares
      interval: hourly
      keepCount: 24
      replicateToHaPeer: true
      enabled: true
2-Node K3s + OpenEBS HA (/etc/axonwall/k3s.yaml)
yaml
k3s:
  enabled: true
  clusterCidr: 10.42.0.0/16
  serviceCidr: 10.43.0.0/16
  nodeIpInterface: lan
  flannelBackend: wireguard-native
  autoJoinHaPeer: true
  openEbs:
    enabled: true
    zfsLocalPvEnabled: true
    zfsLocalPvPool: tank
    replicatedStorageEnabled: true
    defaultReplicaCount: 2
    replicatedBackingPool: tank/openebs-replicated
Export HA-Resilient Kubeconfig Bound to Floating VRRP VIP
bash
# Rewrite ~/.kube/config server endpoint to the floating LAN VRRP VIP (e.g. https://10.0.10.1:6443)
# so kubectl and helm survive control-plane node failover without reconnecting:
axonwall-kubeconfig --vip
# Or via axonwallctl:
axonwallctl kubeconfig --vip
8

Test Live Failover & Graceful Maintenance Mode

Before placing the cluster into production, validate both graceful maintenance demotion and abrupt hardware failover while keeping a continuous SSH session and ping running from a LAN workstation through the 10.0.10.1 floating gateway VIP:

Live Failover & Preemption Validation
bash
# From a LAN client routed through VIP 10.0.10.1, start an active SSH & ping test:
ping -i 0.2 1.1.1.1

# TEST A — Graceful Maintenance / Planned Failover (on Host 1 / axonwall-fw01):
# Set "maintenanceMode: true" under ha: in /etc/axonwall/ha.yaml and run "axonwallctl apply"
# (or click "Force Failover to Peer" -> "Save to ha.yaml" in the /ha Web Console).
# Keepalived lowers effective priority and shifts VIPs + conntrack state to Host 2 with 0 packet loss.

# Verify Host 2 (axonwall-fw02) now owns 198.51.100.1 and 10.0.10.1:
ssh admin@10.0.10.3 "ip -br addr show dev eth0; ip -br addr show dev eth1"

# TEST B — Unplanned Link / Power Loss Simulation (on Host 1 / axonwall-fw01):
# Bring down the WAN link or stop keepalived on Host 1:
sudo ip link set eth0 down
# Host 2 transitions to MASTER within 1000ms, commits conntrackd cache, and active TCP/SSH stays up.

# Restore Host 1 WAN link:
sudo ip link set eth0 up

Cluster PKI, Mutual TLS & Signed State Payloads

Axonwall never transmits configuration state or SQLite WAL frames in cleartext once enrolled. The Go daemon (axonwalld) maintains a dedicated ECDSA P-256 High Availability PKI hierarchy stored in /var/lib/axonwall/secrets/ha-pki/:

Automated CSR Enrollment & Shared Identity

The Primary node generates a 10-year ECDSA P-256 Root CA (ca.crt) and node certificate (node.crt) bound to the cluster's sync IP Subject Alternative Names (SANs). On its first poll, the Backup node submits a CSR over the token-authenticated bootstrap channel to /api/v1/sync/pki-enroll, installs the signed bundle and OpenLDAP syncrepl credentials, and switches to mutual TLS (mtls).

Ed25519 / ECDSA Payload Signing & Safe Cert Rotation

Every /api/v1/sync/state, DHCP lease bundle, and /api/v1/sync/db response on :9876 carries both an X-Checksum-SHA256 digest and a detached Ed25519/ECDSA X-Axonwall-Payload-Signature header. When requireSignedPayloads: true is enabled, unsigned or tampered payloads are strictly rejected before touching disk. Administrators can rotate leaf hub/sync certificates without invalidating Operator-CA trust anchors using sudo axonwall-rotate-hub-cert.

Keepalived VRRP v2/v3 Clustering

Keepalived manages virtual IP address failover according to RFC 5798. Axonwall groups all configured VRRP instances into a unified vrrp_sync_group VG1 so that if either the WAN uplink (eth0) or LAN trunk (eth1) loses carrier, all virtual IPs fail over together—preventing asymmetric blackhole routing.

VRRP Failover Triggers

  • Loss of physical link carrier on WAN or LAN VRRP interfaces
  • Unresponsive axonwalld health probe (chk_axonwalld)
  • Manual maintenance failover (maintenanceMode: true or Role switch in /ha)
  • Power loss or kernel watchdog reset on the active Master

Transition Orchestration Hook

On state change, Keepalived runs conntrackd -c/-f/-t and invokes /usr/local/bin/axonwall-ha-notify, which notifies axonwalld over the root-authenticated Unix socket /run/axonwall/ha.sock to promote verified state replicas, flip OpenZFS pool readonly flags, and start or stop headscale to prevent dual-active split-brain.

conntrackd Stateful Replication Engine

Without state synchronization, a VRRP failover forces all active TCP connections to reset, cutting off active SSH sessions, dropping VoIP calls, and aborting file downloads. Axonwall integrates conntrackd to replicate the kernel Netfilter state table seamlessly across eth3.

FTFW Protocol (Fault-Tolerant Framework)

Unlike BSD pfsync which uses simple unacknowledged UDP broadcasts, the Netfilter FTFW protocol implements sequence-numbered state transactions, explicit sequence acknowledgment, and reliable retransmission of lost state deltas over either multicast (225.0.0.50:3780) or point-to-point unicast UDP.

/etc/conntrackd/conntrackd.conf (Generated by axonwalld)
text
Sync {
    Mode FTFW {
        DisableExternalCache Off
        CommitTimeout 1800
        PurgeTimeout 5
    }
    
    Multicast {
        IPv4_address 225.0.0.50
        IPv4_interface 10.0.99.1
        Interface eth3
        SndSocketBuffer 1249280
        RcvSocketBuffer 1249280
        Checksum on
    }
}

General {
    HashSize 32768
    HashLimit 131072
    LogFile on
    Syslog on
    LockFile /var/lock/conntrack.lock
    UNIX {
        Path /var/run/conntrackd.ctl
        Backlog 20
    }
    NetlinkBufferSize 2097152
    NetlinkBufferSizeMaxGrowth 8388608
    Filter From Userspace {
        Protocol Accept {
            TCP
            UDP
            ICMP
        }
        Address Ignore {
            IPv4_address 127.0.0.1
        }
    }
}

SQLite WAL, Chrony NTP, OpenLDAP, ZFS & K3s Replication

Axonwall extends High Availability beyond L3/L4 packet filtering so stateful control-plane services survive node failure without data loss or split-brain corruption:

SQLite Hot VACUUM INTO & State Replica Promotion

The Primary node creates consistent lock-free hot snapshots via VACUUM INTO (and WAL checkpointing) of allowed databases under /var/lib/axonwall/ (such as /var/lib/axonwall/headscale/db.sqlite) and streams them with SHA-256 and Ed25519/ECDSA signature verification every syncIntervalSeconds seconds. Before replacing a live file, the Backup node validates PRAGMA integrity_check, preserves service UID/GID ownership, and stages state.json.ha-replica for atomic promotion upon MASTER takeover.

Chrony Symmetric NTP & OpenLDAP N-Way syncrepl

OpenLDAP N-Way Multi-Provider syncrepl over ldaps://:636 relies on Change Sequence Numbers (CSNs) with microsecond timestamps. Axonwall automatically writes /etc/chrony/conf.d/axonwall-ha.conf with symmetric peering (peer <peerIp> iburst) on eth3 so both nodes maintain sub-millisecond clock alignment even if upstream internet NTP servers are unreachable.

Incremental OpenZFS Send/Recv

Datasets and snapshot policies marked with replicateToHaPeer: true in /etc/axonwall/storage.yaml are snapshotted (@ha-sync-<ns>) and streamed via zfs send -w -I | zfs recv over /api/v1/sync/zfs to the Backup node. On the Backup node, replicated pools stay readonly=on until Keepalived promotes the node to MASTER.

Embedded-etcd K3s HA Join & Mayastor Volumes

When autoJoinHaPeer: true is set in /etc/axonwall/k3s.yaml, the Backup node securely fetches the K3s embedded-etcd join token from /api/v1/sync/k3s-token over mTLS, joins the cluster over the WireGuard Flannel mesh with axonwall-replicated-ha OpenEBS Mayastor NVMe-oF volumes, and exposes an HA-resilient kubeconfig via axonwall-kubeconfig --vip (axonwallctl kubeconfig --vip).

Failover Latency & Connection Continuity

During an abrupt power failure on the Master appliance, the Backup node detects the missed VRRP advertisement within 1000ms, transitions to MASTER, and flushes the external state cache directly into the Linux Netfilter kernel table:

Protocol / Session TypeBehavior on Legacy BSD FirewallsBehavior on Axonwall HA
Active TCP (SSH, HTTPS, Database)Broken on packet drop / retransmission limitZero-drop; seamlessly continues without RST
VoIP Calls (SIP Signaling / RTP Audio)Audio drops; SIP session terminatesSub-second jitter; call remains connected
WireGuard Site-to-Site TunnelsRequires 30s re-handshakeInstantaneous handover; zero packet drop
Sysctl Performance Tuning
Axonwall automatically tunes net.netfilter.nf_conntrack_tcp_timeout_established = 86400 and expands nf_conntrack_max = 2097152 to handle enterprise connection tables up to 2 million concurrent flows without memory thrashing.