Zero-Instrumentation Kernel Telemetry • eBPF & K3s • v0.13.0

eBPF Infrastructure & Kubernetes Observability

Achieving deep observability across modern infrastructure is essential—yet traditional approaches require invasive application SDK instrumentation and resource-heavy sidecar proxies. Axonwall leverages CO-RE eBPF to gather metrics, logs, and traces directly inside the Linux kernel, tracking container network calls and filesystem operations with sub-microsecond overhead.

1. Gathering Metrics, Logs & Traces Without Application Overhead

Conventional observability stacks force engineering teams to modify application code with language-specific SDKs and inject proxy/logging sidecars into every Kubernetes pod. Every network packet or log line traverses multiple user/kernel context switches, consuming 8–15% of cluster CPU and hundreds of megabytes of RAM per workload.

With Extended Berkeley Packet Filter (eBPF), sandboxed, verifier-checked bytecode runs directly at Ring-0 hook points (`XDP`, `tc`, `kprobes`, `fentry/fexit`, `tracepoints`, and `BPF_LSM`). Because every container, VM, and host daemon shares the same Linux kernel, eBPF observes the entire stack uniformly:

High-Cardinality Metrics

Counters and latency histograms are aggregated in-place inside BPF_MAP_TYPE_LRU_HASH, BPF_MAP_TYPE_HASH, and BPF_MAP_TYPE_PERCPU_ARRAY maps. User space scrapes pre-summarized golden signals (throughput, TCP RTT, retransmits, VFS & OpenZFS ARC/ZIL IOPS) without copying raw packets.

Context-Rich Kernel Logs

Structured security and network events stream asynchronously through a zero-copy BPF_MAP_TYPE_RINGBUF. Each event carries kernel-verified process metadata (pid, uid, cgroup_id, and unspoofable (s_dev, i_ino) inode identity).

Zero-Code Distributed Traces

Socket-level eBPF programs (sock_ops, sk_msg) and user-space TLS uprobes correlate pod-to-pod HTTP/gRPC/DNS calls and storage I/O spans across polyglot microservices with zero SDK rewrites.

2. Tracking Container Network Calls & Filesystem Operations

Containers are isolated using Linux control groups (cgroups v2) and namespaces (mnt, pid, net). By reading kernel task and cgroup structures via BTF Compile Once – Run Everywhere (CO-RE), Axonwall attributes every network socket and filesystem operation to its exact container and Kubernetes pod:

Container Network Call Tracking

  • • Dual-Stack & QinQ VLAN XDP Wire-Speed Accounting: SEC("xdp") peels 2-layer 802.1Q / 802.1ad VLAN headers and tracks IPv4 (flow_map 131K LRU) + IPv6 (flow6_map 65K LRU) 5-tuples and monotonic TTL XDP_DROP blocklists before sk_buff allocation.
  • • Socket Connect & RTT Telemetry: Hooks on tcp_v4_connect and tcp_rcv_established record container cgroup_id, destination service IP:port, and handshake RTT.
  • • Retransmission & Drop Reasons: Tracepoints on tcp_retransmit_skb and kfree_skb expose exact kernel drop reasons (`SKB_DROP_REASON_*`).

Container Filesystem & VFS Operations

  • • VFS & OpenZFS ARC/L2ARC/ZIL Telemetry: BTF fentry/vfs_read, fexit/vfs_write, and vfs_fsync pair with /proc/spl/kstat/zfs ARC/L2ARC hit ratios, ZIL/SLOG sync commits, and per-pool ZIO queues.
  • • Unspoofable (s_dev, i_ino) Identity: Resolves task->mm->exe_file->f_inode so renamed binaries or symlinks inside containers cannot evade attribution.
  • • Sensitive File Access Auditing: Tetragon security_file_open selectors detect unexpected reads of TLS keys, service-account tokens, or `/etc/shadow`.
CO-RE eBPF Container & Executable Attribution Pattern (apps/axonwalld/ebpf/lsm_guard.c)
c
struct lsm_exe_id {
    __u64 dev;
    __u64 ino;
};

static __always_inline bool get_current_exe_id(struct lsm_exe_id *out) {
    struct task_struct *task = (struct task_struct *)bpf_get_current_task_btf();
    if (!task)
        return false;

    struct file *exe_file = BPF_CORE_READ(task, mm, exe_file);
    if (!exe_file)
        return false;

    struct inode *inode = BPF_CORE_READ(exe_file, f_inode);
    if (!inode)
        return false;

    // Kernel super_block s_dev encodes (MAJOR << 20) | MINOR
    __u32 s_dev = BPF_CORE_READ(inode, i_sb, s_dev);
    __u64 i_ino = BPF_CORE_READ(inode, i_ino);

    out->dev = (__u64)s_dev;
    out->ino = i_ino;
    return true;
}

3. Troubleshooting Kubernetes (K3s) Environments with eBPF

In Kubernetes clusters, distroless containers lack shells, `tcpdump`, `strace`, or `netstat`, and ephemeral pods vanish before operators can inspect them. Axonwall combines host-level eBPF telemetry with the Vigilant Octo Waffle (VOW) observability stack (Cilium Hubble, Tetragon, Kubeshark, and OpenTelemetry) to solve four common cluster failure modes:

1. DNS Lookup Amplification & Drops

Inspect per-pod DNS queries and `RCode` responses (`NXDOMAIN`, `SERVFAIL`) via cgroup socket filters and correlate UDP socket receive-buffer drops without sidecars.

2. CNI Overlay MTU & TCP Retransmissions

Use tracepoint/tcp/tcp_retransmit_skb to spot PMTU blackholes and congestion window drops across WireGuard-encrypted K3s node links.

3. PVC Storage Latency (OpenEBS ZFS-LocalPV)

Attribute slow vfs_write and vfs_fsync calls to specific stateful pods (PostgreSQL, etcd, ClickHouse) to separate disk contention from network latency.

4. Zero-Syscall & Kernel Crypto Containment

Enforce all 5 CO-RE BPF-LSM hooks—SEC("lsm/socket_bind") (Copy Fail CVE-2026-31431), SEC("lsm/bpf"), SEC("lsm/uring_sqpoll"), SEC("lsm/bpf_token_create"), and SEC("lsm/bpf_token_cmd")—across all pods.

K3s eBPF Troubleshooting CLI Workflow
bash
# 1. Inspect Dual-Stack XDP counters, 5-hook BPF-LSM & Linux 6.9+ BPF_TOKEN mounts
axonwallctl ebpf overview
axonwallctl ebpf tokens

# 2. Observe real-time L7 DNS and HTTP errors in K3s via Cilium Hubble
hubble observe --namespace vow-system --verdict DROPPED,ERROR --follow

# 3. Inspect Axonwall XDP flow_map/flow6_map and BPF-LSM allowlist maps
bpftool prog show
bpftool map dump name allowlist

# 4. Stream Tetragon container process & VFS file-access events
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f

4. Coroot-Grade eBPF APM, Automated RCA & Continuous Profiling

Axonwall integrates Coroot-inspired eBPF Application Performance Monitoring (APM) both natively inside /reporting/ebpf and as a toggleable plugin (coroot-ebpf-apm / VOW catalog coroot). Using guardrailed observability.ebpfApm toggles, operators can enable or disable higher-overhead probes on demand without restarting workloads:

1. Auto-Discovered Service Dependency Map

Hooks tcp_connect, inet_csk_accept, and sock_sendmsg to construct a real-time topology graph of every host daemon and K3s pod alongside live connection states, RTT, and TCP retransmission rates.

2. L7 Protocol RED Dissection + TLS Uprobes

Automatically parses HTTP/1.1, HTTP/2, gRPC, PostgreSQL, Redis, ClickHouse, DNS, and Kafka payloads at socket boundaries to compute per-endpoint Rate, Errors, and Duration (`p50`/`p95`/`p99`). Optional uprobe/SSL_read, uprobe/SSL_write, and Go crypto/tls hooks inspect TLS-encrypted traffic before ciphertext leaves user space—without MITM certificates.

3. Automated Root-Cause Analysis (RCA) Engine

Deterministic inspection rules evaluate SLO availability, cgroup CFS CPU throttling, OOM kills / memory pressure, TCP retransmit spikes, OpenEBS ZFS-LocalPV vfs_fsync latency, and BPF-LSM security blocks—pinpointing the exact root cause of latency anomalies.

4. 99 Hz Continuous Stack-Trace FlameGraphs

Samples kernel + user-space stack traces via perf_event and BPF_MAP_TYPE_STACK_TRACE across On-CPU cycles, Disk/ZFS I/O wait, Heap/ARC memory allocations, and Off-CPU mutex contention.

/etc/axonwall/system.yaml — Complete ObservabilitySchema (observability: & coroot-ebpf-apm)
yaml
plugins:
  - id: coroot-ebpf-apm
    enabled: true

observability:
  metrics:
    enabled: true
    retentionDays: 30
    remoteWriteUrl: ""
  flowExport:
    enabled: false
    protocol: ipfix
    collectorAddress: ""
  alerting:
    enabled: true
    webhookUrl: ""
  clickhouse:
    enabled: false
    endpoint: http://127.0.0.1:8123
    database: axonwall_telemetry
  ebpfApm:
    enabled: true
    serviceMapEnabled: true
    l7ProtocolDissection: true
    tlsUprobesEnabled: false
    continuousProfilingEnabled: true
    profilingSampleHz: 99
    automatedRcaInspections: true
    xdpDualStackVlanEnabled: true
    zfsArcZilTelemetryEnabled: true

Linux 6.9+ `BPF_TOKEN_CREATE` Unprivileged Delegation & Dual-Stack XDP

Informed by Tekromancy's kernel engineering research on Linux 6.9+ `BPF_TOKEN_CREATE` & User Namespace eBPF Delegation, Axonwall bridges multi-tenant container isolation (`CLONE_NEWUSER`) with high-speed packet processing—allowing rootless K3s hostUsers: false pods and unprivileged LXC containers to load verified networking eBPF programs without granting host CAP_BPF or CAP_SYS_ADMIN:

1. Four-Pillar `bpffs` Mount Delegation

axonwalld mounts isolated token instances under /run/axonwall/bpffs-tokens/<profile> with explicit delegate_cmds, delegate_progs, delegate_maps, and delegate_attachs options. Inside the container, setting LIBBPF_BPF_TOKEN_PATH=/sys/fs/bpf enables libbpf and cilium/ebpf to mint `BPF_TOKEN_CREATE` file descriptors transparently.

2. Ring-0 `bpf_probe_read_kernel()` Isolation Veto

Even inside a User Namespace, eBPF tracing programs execute in shared Ring-0 address space where bpf_probe_read_kernel() could read host credentials or sibling container secrets. Axonwall's CO-RE BPF-LSM hooks (SEC("lsm/bpf_token_create") & SEC("lsm/bpf_token_cmd")) unconditionally block kprobe, tracepoint, perf_event, tracing, struct_ops, and lsm with -EPERM.

3. Dual-Stack IPv4/IPv6 + 802.1Q/802.1ad QinQ XDP

flow_tracker.c peels up to two nested 802.1Q / 802.1ad VLAN headers and tracks both IPv4 (flow_map, 131K LRU) and IPv6 (flow6_map, 65K LRU) 5-tuples, TCP SYN/RST flags, and per-CPU verdict counters at ~18 ns/packet.

4. LPM CIDR Subnet `XDP_DROP` & Ring-0 TCP SYN-Flood Shield

Dual-stack exact-host maps (blocklist_map / blocklist6_map) and BPF_MAP_TYPE_LPM_TRIE prefix maps (cidr4_blocklist_map / cidr6_blocklist_map) store monotonic bpf_ktime_get_ns() expiration timestamps in-kernel, paired with a per-source 1-second sliding-window Ring-0 TCP SYN-flood rate limiter (syn_rate_map / syn6_rate_map) that drops volumetric SYN floods before sk_buff allocation.

axonwallctl ebpf — Ring-0 Dataplane, XDP Fast-Drop & BPF_TOKEN CLI
bash
# Unified Ring-0 eBPF, BPF_TOKEN_CREATE, KVM/LXC, K3s & OpenZFS ARC/ZIL overview
axonwallctl ebpf overview

# Inspect Linux 6.9+ BPF_TOKEN kernel support, presets, and active bpffs mounts
axonwallctl ebpf tokens

# Program a line-rate dual-stack XDP_DROP lease (IPv4 or IPv6) with 1-hour TTL
axonwallctl ebpf xdp-drop 198.51.100.77 3600
axonwallctl ebpf xdp-blocks
axonwallctl ebpf xdp-unblock 198.51.100.77

# Render a K3s UserNS (hostUsers: false) DaemonSet manifest with LIBBPF_BPF_TOKEN_PATH=/sys/fs/bpf
axonwallctl ebpf render-k3s ebpf-sensor k3s-net-safe networking-safe

5. Live eBPF Visualizations & Kernel BPF Inventory

GET /v1/ebpf/overview (and its Web Console proxy GET /api/v1/ebpf/overview) exposes CollectKernelBpfInventory() (bpfInventory) alongside live Ring-0 XDP counters, BPF-LSM blocks, cross-duty storage/hypervisor telemetry, and Blackbox threat sketches—powering six live visualizers across /reporting/ebpf, /reporting/flows, and /security/threat-harvester:

EbpfDataplaneCharts

Six live charts: Real-Time XDP Ring-0 Verdict Rate, Per-CPU XDP Core Load, Protocol, Dual-Stack & VLAN Mix, Top Destination Services by XDP Bytes, XDP Blocklist Impact by Source / CIDR, and Per-Source TCP SYN Rate Meter.

EbpfFlowTopologyMap

Interactive live XDP flow topology graph linking ingress NICs/VLANs, active IPv4/IPv6 tuples, blocked peers, and downstream destinations.

EbpfKernelInventoryPanel

Live kernel enumeration of loaded BPF programs (ID, type, JIT & translated bytes, verified instructions, BTF status, cumulative runCount/runTimeNs) and BPF maps (key/value/max entries, memlock bytes, frozen flag).

EbpfCrossDutyVisualizer

Correlates OpenZFS SPL kstat ARC/L2ARC/ZIL throughput, KVM vnet/tap NIC queues, and K3s pod eBPF token metrics in a unified view.

EbpfSecurityVisualizer & ThreatSketchWaveform

Visualizes 5-hook CO-RE BPF-LSM enforcement events, Linux 6.9+ BPF_TOKEN mounts, and signed packet-size / microsecond inter-arrival-time waveforms for Blackbox Threat-LLM episodes.

FlowsLiveVisualStrip

Heads-up protocol, direction, verdict, and burst Sparkline strip embedded above the live packet & flow table at /reporting/flows.

6. Kernel Packet Stream Architecture (NFLOG & XDP)

Traditional BSD firewalls require administrators to start a packet capture, wait for an incident to occur, download an unwieldy .pcap file, and open Wireshark on a workstation. Axonwall taps directly into the Linux netfilter subsystem via NFLOG and eBPF:

Packet Capture & eBPF Ring-Buffer Dataflow
text
[ Network Interface / K3s CNI (wan0 / lan0 / cni0) ]
                 |
                 v
     [ XDP Hook (xdp_flow_tracker) ]  <-- Pre-sk_buff 5-tuple metrics & blocklist drop
                 |
                 v
       [ Linux Kernel Netfilter ]
                 |
      NFLOG Group / eBPF Tap + BPF-LSM RingBuf
                 |
                 v
    [ axonwalld Packet & LSM Streamer ] <-- Ring-buffered zero-copy memory ring
                 |
                 v
      [ Unix Domain Socket ]
                 |
                 v
  [ Axonwall Web Controller (SSE) ]
   GET /api/v1/events/packets
                 |
                 v
  [ Live Browser Terminal / UI ]        <-- Instant interactive flow & security stream

7. Deep Protocol Dissector

The embedded Go daemon dissector analyzes raw Ethernet frames at wire-speed and parses packet headers without writing files to disk:

Layer 2 (Ethernet)

Source & Destination MAC addresses, VLAN tags (802.1Q), and EtherType decoding.

Layer 3 (IP / IPv6)

Source/Destination IPs, TTL/Hop Limit, Type of Service (TOS), and packet length.

Layer 4 (TCP / UDP / ICMP)

Ports, sequence numbers, TCP flags (SYN, ACK, FIN, RST, PSH), and ICMP type/code.

8. Server-Sent Events (SSE) API

Clients stream live packet telemetry over HTTP/2 using Server-Sent Events. The stream supports server-side filtering by interface, protocol, IP address, and port:

Streaming Example
bash
# Stream live TCP packets on interface eth0 with SYN or RST flags
curl -N "https://192.168.1.1:2966/api/v1/events/packets?iface=eth0&proto=tcp"

Example Event Payload (JSON)

SSE Packet Event
json
{
  "timestamp": "2026-10-13T07:15:22.408Z",
  "interface": "wan0",
  "direction": "in",
  "ethernet": {
    "src": "00:1a:2b:3c:4d:5e",
    "dst": "52:54:00:12:34:56",
    "type": "IPv4"
  },
  "ip": {
    "version": 4,
    "src": "198.51.100.24",
    "dst": "192.168.1.1",
    "ttl": 54,
    "proto": "TCP",
    "length": 60
  },
  "tcp": {
    "srcPort": 52414,
    "dstPort": 443,
    "seq": 284192041,
    "flags": ["SYN"]
  },
  "action": "pass"
}

9. VictoriaMetrics, ClickHouse & Insight Flow Analytics

In addition to real-time eBPF and NFLOG streaming, historical time-series metrics are stored in an embedded, resource-efficient VictoriaMetrics engine and optional ClickHouse columnar store. This replaces legacy RRDtool graphs with high-speed drill-down analytics for bandwidth usage, top talkers, K3s pod flows, and application protocols.

Web Console Dashboards
Explore live eBPF observability in your console under Reporting > eBPF & K3s Observability (/reporting/ebpf), Reporting > Live Flow Monitor (/reporting/flows), and Diagnostics > Packet Tracer (/diagnostics/packet-tracer).