Container Orchestration & Cloud-Native Storage • v0.13.0

Kubernetes (K3s) & OpenEBS Distributed Storage

Axonwall provides modern, lightweight K3s alongside enterprise-grade NAS storage configured via /etc/axonwall/k3s.yaml. Both kubectl, interactive TUI k9s (v0.51.0, alias k9, bundled inside core-k3s alongside k3s), and helm v3 are bundled out-of-the-box, paired with our full Browser K9s Live Explorer (/k3s/explorer). Run containerized workloads directly on your appliance with encrypted WireGuard flannel networking, automated 2-node embedded-etcd HA cluster join, and OpenEBS replicated persistent volumes.

Barebones K3s + Storage Edition (k3s-storage): Deploying a dedicated headless K3s + OpenZFS/OpenEBS node without firewall/NAS bloat? Select the k3s-storage edition directly from the unified ISO bootloader or installer TUI (see Barebones K3s + Storage Guide) and pair it with our Hardened Linux 6.12 LTS Kernel & Custom Kernel Studio.

Embedded K3s Architecture

K3s is packaged as a single binary with zero external dependencies, embedded SQLite/etcd consensus, and native Containerd. Axonwall manages K3s through the privileged Go daemon (axonwalld), which generates declarative control plane manifests and reconciles cluster state.

WireGuard Native Flannel

Pod-to-pod cross-node traffic is automatically encrypted using Linux kernel WireGuard (flannelBackend: "wireguard-native"), requiring zero overlay CPU overhead.

Automatic 2-Node HA Join

When autoJoinHaPeer: true is enabled, the secondary HA node connects to the primary node over mTLS, fetches the cluster join token via /api/v1/sync/k3s-token into /var/lib/axonwall/k3s-join.token, and joins the cluster automatically.

VIP Ingress Integration

K3s workloads bind directly to Axonwall VRRP Virtual IPs, ensuring that if a physical node experiences hardware failure, traffic routes to the surviving node in under one second.

Declarative K3s, Cilium eBPF CNI, Etcd S3 Snapshots & OpenEBS (/etc/axonwall/k3s.yaml)
yaml
k3s:
  enabled: true
  clusterCidr: 10.42.0.0/16
  serviceCidr: 10.43.0.0/16
  nodeIpInterface: lan
  cniPlugin: cilium-ebpf          # 'flannel' (WireGuard) or 'cilium-ebpf' (kube-proxy replacement)
  ciliumHubbleEnabled: true       # L7 eBPF pod flow observability when cniPlugin=cilium-ebpf
  flannelBackend: wireguard-native
  disableTraefik: false
  autoJoinHaPeer: true
  enableUserNamespaces: true
  etcdSnapshotScheduleCron: "0 */6 * * *"
  etcdSnapshotRetention: 10
  etcdS3Backup:
    enabled: true
    endpoint: s3.axonwall.lan:8333
    bucket: axonwall-k3s-etcd
  bpfTokenProfiles:
    - name: k3s-net-safe
      preset: networking-safe
  openEbs:
    enabled: true
    zfsLocalPvEnabled: true
    zfsLocalPvPool: tank
    replicatedStorageEnabled: true
    defaultReplicaCount: 2
    replicatedBackingPool: tank/openebs-replicated

Cilium eBPF CNI Mode & Embedded Etcd Snapshots

Choose between lightweight flannel (WireGuard-native) for minimalist edge appliances or full cniPlugin: "cilium-ebpf" for high-density clusters, paired with automated embedded-etcd disaster recovery:

Kube-Proxy-Less Cilium eBPF + Hubble

When cniPlugin: "cilium-ebpf" is selected, axonwalld automatically compiles K3s with flannel-backend: none, disable-network-policy: true, and disable-kube-proxy: true, deploying Cilium HelmChartConfig with socket-level eBPF Service load-balancing (kubeProxyReplacement: true), WireGuard encryption, and Hubble L7 observability.

On-Demand & S3 Embedded Etcd Snapshots

Trigger point-in-time control-plane snapshots (axonwallctl k3s etcd-save [name]), enumerate local & S3-replicated snapshots (axonwallctl k3s etcd-snapshots), and execute guarded cluster restores (axonwallctl k3s etcd-restore <name>) from the Web Console (/k3s/cluster), CLI, or mcp_manage_k3s_cluster.

OpenEBS CSI StorageClasses

Axonwall provisions two production-grade OpenEBS StorageClasses out of the box:

axonwall-zfs-localLocal PV

Provisions dynamic volumes directly on your host ZFS pool (zfs.csi.openebs.io). Delivers bare-metal NVMe throughput with native ZFS compression, snapshots, and quota enforcement.

axonwall-replicated-haDefault CSI

Synchronous 2-node block replication over NVMe-oF (io.openebs.csi-mayastor). Every write is mirrored across both HA Axonwall appliances in real time. If node 1 drops, stateful workloads restart on node 2 instantly with zero data loss.

Sample PersistentVolumeClaim (PVC)
yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: axonwall-replicated-ha
  resources:
    requests:
      storage: 100Gi

Kubeconfig Permissions, HA VIP Sync & Remote Workstation Merge

By default, upstream K3s writes /etc/rancher/k3s/k3s.yaml with 0600 root:root permissions, blocking non-root administrators and the Web Console API. When axonwalld provisions K3s, it enforces group-readable kubeconfig access (write-kubeconfig-mode: "0640", write-kubeconfig-group: "axonwall-api") and ships first-class kubeconfig sync utilities:

1. Local Admin Sync

Run axonwall-kubeconfig (or axonwallctl kubeconfig) as your SSH admin user to copy and chmod ~/.kube/config (0600) for immediate kubectl and helm v3 usage without sudo.

2. HA Floating VIP Endpoint (--vip)

Pass --vip to automatically rewrite server: https://127.0.0.1:6443 to your active Keepalived VRRP Virtual IP (which K3s automatically includes in its API server tls-san list), keeping kubectl online across Master/Backup failover.

3. Remote Laptop SSH Merge

From a developer workstation, run ./scripts/update-kubeconfig.sh over SSH to fetch the cluster certificate and non-destructively merge the axonwall context into your local multi-cluster ~/.kube/config.

Local, HA VIP & Remote Workstation Kubeconfig Sync
bash
# On the Axonwall appliance: launch k9s directly over SSH (KUBECONFIG auto-detected) or sync ~/.kube/config
axonwall-kubeconfig
kubectl get nodes -o wide
k9s -A

# On a 2-Host HA cluster: bind kubeconfig server URL to the floating VRRP VIP
axonwall-kubeconfig --vip
# Equivalent axonwallctl subcommand (prints or writes VIP-rewritten kubeconfig):
axonwallctl kubeconfig --vip

# From your remote Linux/macOS workstation: fetch & merge over SSH
./scripts/update-kubeconfig.sh user@10.80.0.202 --vip
kubectl --context axonwall get pods -A
k9s --context axonwall

Browser K9s Live Explorer, Pulse & Topology Map

Open /k3s/explorer in the Web Console to access full browser-native K9s functionality: keyboard-driven navigation (:po, :svc, :ing, :pulse, :topo, / filter, 0..9 namespaces), Cluster Pulse CPU/Memory/Pod Recharts visualizations, an interactive Ingress → Service → Workload → Pod → Node / PVC SVG Topology Map, live SSE log streaming, interactive container exec, editable YAML with server dry-run, and admin Secret decoding.

Node Roles & Taints Modal (/k3s/cluster)

Manage cluster scheduling boundaries directly from /k3s/cluster via POST /v1/k3s/nodes/scheduling. Apply one-click Edge Node taint/role presets, customize node labels and NoSchedule/NoExecute taints, and merge-patch node specs without manual kubectl taint invocations so edge gateways host only essential network pods.

Linux 6.9+ User Namespaces (`hostUsers: false`) & `BPF_TOKEN` Pods

Setting enableUserNamespaces: true in /etc/axonwall/k3s.yaml compiles UserNamespacesSupport=true across kube-apiserver and kubelet in /etc/rancher/k3s/config.yaml. Combined with bpfTokenProfiles, K3s workloads can run inside isolated User Namespaces (hostUsers: false) and mount a delegated bpffs directory at /sys/fs/bpf with LIBBPF_BPF_TOKEN_PATH=/sys/fs/bpf—loading XDP/TC/Socket networking eBPF programs while Axonwall BPF-LSM blocks kprobe/tracepoint/lsm Ring-0 memory reads:

Rootless K3s UserNS DaemonSet Using Linux 6.9+ BPF_TOKEN (axonwallctl ebpf render-k3s)
yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ebpf-packet-sensor
  namespace: default
spec:
  selector:
    matchLabels:
      app: ebpf-packet-sensor
  template:
    metadata:
      labels:
        app: ebpf-packet-sensor
    spec:
      hostUsers: false
      containers:
        - name: ebpf-packet-sensor
          image: ghcr.io/cilium/ebpf-sensor:latest
          env:
            - name: LIBBPF_BPF_TOKEN_PATH
              value: /sys/fs/bpf
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
              add: ["BPF", "NET_ADMIN"]
          volumeMounts:
            - name: delegated-bpffs
              mountPath: /sys/fs/bpf
      volumes:
        - name: delegated-bpffs
          hostPath:
            path: /run/axonwall/bpffs-tokens/k3s-net-safe
            type: Directory