eBPF Infrastructure & Kubernetes Observability
Achieving deep observability across modern infrastructure is essential—yet traditional approaches require invasive application SDK instrumentation and resource-heavy sidecar proxies. Axonwall leverages CO-RE eBPF to gather metrics, logs, and traces directly inside the Linux kernel, tracking container network calls and filesystem operations with sub-microsecond overhead.
1. Gathering Metrics, Logs & Traces Without Application Overhead
Conventional observability stacks force engineering teams to modify application code with language-specific SDKs and inject proxy/logging sidecars into every Kubernetes pod. Every network packet or log line traverses multiple user/kernel context switches, consuming 8–15% of cluster CPU and hundreds of megabytes of RAM per workload.
With Extended Berkeley Packet Filter (eBPF), sandboxed, verifier-checked bytecode runs directly at Ring-0 hook points (`XDP`, `tc`, `kprobes`, `fentry/fexit`, `tracepoints`, and `BPF_LSM`). Because every container, VM, and host daemon shares the same Linux kernel, eBPF observes the entire stack uniformly:
Counters and latency histograms are aggregated in-place inside BPF_MAP_TYPE_LRU_HASH, BPF_MAP_TYPE_HASH, and BPF_MAP_TYPE_PERCPU_ARRAY maps. User space scrapes pre-summarized golden signals (throughput, TCP RTT, retransmits, VFS & OpenZFS ARC/ZIL IOPS) without copying raw packets.
Structured security and network events stream asynchronously through a zero-copy BPF_MAP_TYPE_RINGBUF. Each event carries kernel-verified process metadata (pid, uid, cgroup_id, and unspoofable (s_dev, i_ino) inode identity).
Socket-level eBPF programs (sock_ops, sk_msg) and user-space TLS uprobes correlate pod-to-pod HTTP/gRPC/DNS calls and storage I/O spans across polyglot microservices with zero SDK rewrites.
2. Tracking Container Network Calls & Filesystem Operations
Containers are isolated using Linux control groups (cgroups v2) and namespaces (mnt, pid, net). By reading kernel task and cgroup structures via BTF Compile Once – Run Everywhere (CO-RE), Axonwall attributes every network socket and filesystem operation to its exact container and Kubernetes pod:
Container Network Call Tracking
- • Dual-Stack & QinQ VLAN XDP Wire-Speed Accounting:
SEC("xdp")peels 2-layer802.1Q/802.1adVLAN headers and tracks IPv4 (flow_map131K LRU) + IPv6 (flow6_map65K LRU) 5-tuples and monotonic TTLXDP_DROPblocklists beforesk_buffallocation. - • Socket Connect & RTT Telemetry: Hooks on
tcp_v4_connectandtcp_rcv_establishedrecord containercgroup_id, destination service IP:port, and handshake RTT. - • Retransmission & Drop Reasons: Tracepoints on
tcp_retransmit_skbandkfree_skbexpose exact kernel drop reasons (`SKB_DROP_REASON_*`).
Container Filesystem & VFS Operations
- • VFS & OpenZFS ARC/L2ARC/ZIL Telemetry: BTF
fentry/vfs_read,fexit/vfs_write, andvfs_fsyncpair with/proc/spl/kstat/zfsARC/L2ARC hit ratios, ZIL/SLOG sync commits, and per-pool ZIO queues. - • Unspoofable (s_dev, i_ino) Identity: Resolves
task->mm->exe_file->f_inodeso renamed binaries or symlinks inside containers cannot evade attribution. - • Sensitive File Access Auditing: Tetragon
security_file_openselectors detect unexpected reads of TLS keys, service-account tokens, or `/etc/shadow`.
struct lsm_exe_id {
__u64 dev;
__u64 ino;
};
static __always_inline bool get_current_exe_id(struct lsm_exe_id *out) {
struct task_struct *task = (struct task_struct *)bpf_get_current_task_btf();
if (!task)
return false;
struct file *exe_file = BPF_CORE_READ(task, mm, exe_file);
if (!exe_file)
return false;
struct inode *inode = BPF_CORE_READ(exe_file, f_inode);
if (!inode)
return false;
// Kernel super_block s_dev encodes (MAJOR << 20) | MINOR
__u32 s_dev = BPF_CORE_READ(inode, i_sb, s_dev);
__u64 i_ino = BPF_CORE_READ(inode, i_ino);
out->dev = (__u64)s_dev;
out->ino = i_ino;
return true;
}3. Troubleshooting Kubernetes (K3s) Environments with eBPF
In Kubernetes clusters, distroless containers lack shells, `tcpdump`, `strace`, or `netstat`, and ephemeral pods vanish before operators can inspect them. Axonwall combines host-level eBPF telemetry with the Vigilant Octo Waffle (VOW) observability stack (Cilium Hubble, Tetragon, Kubeshark, and OpenTelemetry) to solve four common cluster failure modes:
Inspect per-pod DNS queries and `RCode` responses (`NXDOMAIN`, `SERVFAIL`) via cgroup socket filters and correlate UDP socket receive-buffer drops without sidecars.
Use tracepoint/tcp/tcp_retransmit_skb to spot PMTU blackholes and congestion window drops across WireGuard-encrypted K3s node links.
Attribute slow vfs_write and vfs_fsync calls to specific stateful pods (PostgreSQL, etcd, ClickHouse) to separate disk contention from network latency.
Enforce all 5 CO-RE BPF-LSM hooks—SEC("lsm/socket_bind") (Copy Fail CVE-2026-31431), SEC("lsm/bpf"), SEC("lsm/uring_sqpoll"), SEC("lsm/bpf_token_create"), and SEC("lsm/bpf_token_cmd")—across all pods.
# 1. Inspect Dual-Stack XDP counters, 5-hook BPF-LSM & Linux 6.9+ BPF_TOKEN mounts axonwallctl ebpf overview axonwallctl ebpf tokens # 2. Observe real-time L7 DNS and HTTP errors in K3s via Cilium Hubble hubble observe --namespace vow-system --verdict DROPPED,ERROR --follow # 3. Inspect Axonwall XDP flow_map/flow6_map and BPF-LSM allowlist maps bpftool prog show bpftool map dump name allowlist # 4. Stream Tetragon container process & VFS file-access events kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f
4. Coroot-Grade eBPF APM, Automated RCA & Continuous Profiling
Axonwall integrates Coroot-inspired eBPF Application Performance Monitoring (APM) both natively inside /reporting/ebpf and as a toggleable plugin (coroot-ebpf-apm / VOW catalog coroot). Using guardrailed observability.ebpfApm toggles, operators can enable or disable higher-overhead probes on demand without restarting workloads:
Hooks tcp_connect, inet_csk_accept, and sock_sendmsg to construct a real-time topology graph of every host daemon and K3s pod alongside live connection states, RTT, and TCP retransmission rates.
Automatically parses HTTP/1.1, HTTP/2, gRPC, PostgreSQL, Redis, ClickHouse, DNS, and Kafka payloads at socket boundaries to compute per-endpoint Rate, Errors, and Duration (`p50`/`p95`/`p99`). Optional uprobe/SSL_read, uprobe/SSL_write, and Go crypto/tls hooks inspect TLS-encrypted traffic before ciphertext leaves user space—without MITM certificates.
Deterministic inspection rules evaluate SLO availability, cgroup CFS CPU throttling, OOM kills / memory pressure, TCP retransmit spikes, OpenEBS ZFS-LocalPV vfs_fsync latency, and BPF-LSM security blocks—pinpointing the exact root cause of latency anomalies.
Samples kernel + user-space stack traces via perf_event and BPF_MAP_TYPE_STACK_TRACE across On-CPU cycles, Disk/ZFS I/O wait, Heap/ARC memory allocations, and Off-CPU mutex contention.
plugins:
- id: coroot-ebpf-apm
enabled: true
observability:
metrics:
enabled: true
retentionDays: 30
remoteWriteUrl: ""
flowExport:
enabled: false
protocol: ipfix
collectorAddress: ""
alerting:
enabled: true
webhookUrl: ""
clickhouse:
enabled: false
endpoint: http://127.0.0.1:8123
database: axonwall_telemetry
ebpfApm:
enabled: true
serviceMapEnabled: true
l7ProtocolDissection: true
tlsUprobesEnabled: false
continuousProfilingEnabled: true
profilingSampleHz: 99
automatedRcaInspections: true
xdpDualStackVlanEnabled: true
zfsArcZilTelemetryEnabled: trueLinux 6.9+ `BPF_TOKEN_CREATE` Unprivileged Delegation & Dual-Stack XDP
Informed by Tekromancy's kernel engineering research on Linux 6.9+ `BPF_TOKEN_CREATE` & User Namespace eBPF Delegation, Axonwall bridges multi-tenant container isolation (`CLONE_NEWUSER`) with high-speed packet processing—allowing rootless K3s hostUsers: false pods and unprivileged LXC containers to load verified networking eBPF programs without granting host CAP_BPF or CAP_SYS_ADMIN:
axonwalld mounts isolated token instances under /run/axonwall/bpffs-tokens/<profile> with explicit delegate_cmds, delegate_progs, delegate_maps, and delegate_attachs options. Inside the container, setting LIBBPF_BPF_TOKEN_PATH=/sys/fs/bpf enables libbpf and cilium/ebpf to mint `BPF_TOKEN_CREATE` file descriptors transparently.
Even inside a User Namespace, eBPF tracing programs execute in shared Ring-0 address space where bpf_probe_read_kernel() could read host credentials or sibling container secrets. Axonwall's CO-RE BPF-LSM hooks (SEC("lsm/bpf_token_create") & SEC("lsm/bpf_token_cmd")) unconditionally block kprobe, tracepoint, perf_event, tracing, struct_ops, and lsm with -EPERM.
flow_tracker.c peels up to two nested 802.1Q / 802.1ad VLAN headers and tracks both IPv4 (flow_map, 131K LRU) and IPv6 (flow6_map, 65K LRU) 5-tuples, TCP SYN/RST flags, and per-CPU verdict counters at ~18 ns/packet.
Dual-stack exact-host maps (blocklist_map / blocklist6_map) and BPF_MAP_TYPE_LPM_TRIE prefix maps (cidr4_blocklist_map / cidr6_blocklist_map) store monotonic bpf_ktime_get_ns() expiration timestamps in-kernel, paired with a per-source 1-second sliding-window Ring-0 TCP SYN-flood rate limiter (syn_rate_map / syn6_rate_map) that drops volumetric SYN floods before sk_buff allocation.
# Unified Ring-0 eBPF, BPF_TOKEN_CREATE, KVM/LXC, K3s & OpenZFS ARC/ZIL overview axonwallctl ebpf overview # Inspect Linux 6.9+ BPF_TOKEN kernel support, presets, and active bpffs mounts axonwallctl ebpf tokens # Program a line-rate dual-stack XDP_DROP lease (IPv4 or IPv6) with 1-hour TTL axonwallctl ebpf xdp-drop 198.51.100.77 3600 axonwallctl ebpf xdp-blocks axonwallctl ebpf xdp-unblock 198.51.100.77 # Render a K3s UserNS (hostUsers: false) DaemonSet manifest with LIBBPF_BPF_TOKEN_PATH=/sys/fs/bpf axonwallctl ebpf render-k3s ebpf-sensor k3s-net-safe networking-safe
5. Live eBPF Visualizations & Kernel BPF Inventory
GET /v1/ebpf/overview (and its Web Console proxy GET /api/v1/ebpf/overview) exposes CollectKernelBpfInventory() (bpfInventory) alongside live Ring-0 XDP counters, BPF-LSM blocks, cross-duty storage/hypervisor telemetry, and Blackbox threat sketches—powering six live visualizers across /reporting/ebpf, /reporting/flows, and /security/threat-harvester:
Six live charts: Real-Time XDP Ring-0 Verdict Rate, Per-CPU XDP Core Load, Protocol, Dual-Stack & VLAN Mix, Top Destination Services by XDP Bytes, XDP Blocklist Impact by Source / CIDR, and Per-Source TCP SYN Rate Meter.
Interactive live XDP flow topology graph linking ingress NICs/VLANs, active IPv4/IPv6 tuples, blocked peers, and downstream destinations.
Live kernel enumeration of loaded BPF programs (ID, type, JIT & translated bytes, verified instructions, BTF status, cumulative runCount/runTimeNs) and BPF maps (key/value/max entries, memlock bytes, frozen flag).
Correlates OpenZFS SPL kstat ARC/L2ARC/ZIL throughput, KVM vnet/tap NIC queues, and K3s pod eBPF token metrics in a unified view.
Visualizes 5-hook CO-RE BPF-LSM enforcement events, Linux 6.9+ BPF_TOKEN mounts, and signed packet-size / microsecond inter-arrival-time waveforms for Blackbox Threat-LLM episodes.
Heads-up protocol, direction, verdict, and burst Sparkline strip embedded above the live packet & flow table at /reporting/flows.
6. Kernel Packet Stream Architecture (NFLOG & XDP)
Traditional BSD firewalls require administrators to start a packet capture, wait for an incident to occur, download an unwieldy .pcap file, and open Wireshark on a workstation. Axonwall taps directly into the Linux netfilter subsystem via NFLOG and eBPF:
[ Network Interface / K3s CNI (wan0 / lan0 / cni0) ]
|
v
[ XDP Hook (xdp_flow_tracker) ] <-- Pre-sk_buff 5-tuple metrics & blocklist drop
|
v
[ Linux Kernel Netfilter ]
|
NFLOG Group / eBPF Tap + BPF-LSM RingBuf
|
v
[ axonwalld Packet & LSM Streamer ] <-- Ring-buffered zero-copy memory ring
|
v
[ Unix Domain Socket ]
|
v
[ Axonwall Web Controller (SSE) ]
GET /api/v1/events/packets
|
v
[ Live Browser Terminal / UI ] <-- Instant interactive flow & security stream7. Deep Protocol Dissector
The embedded Go daemon dissector analyzes raw Ethernet frames at wire-speed and parses packet headers without writing files to disk:
Source & Destination MAC addresses, VLAN tags (802.1Q), and EtherType decoding.
Source/Destination IPs, TTL/Hop Limit, Type of Service (TOS), and packet length.
Ports, sequence numbers, TCP flags (SYN, ACK, FIN, RST, PSH), and ICMP type/code.
8. Server-Sent Events (SSE) API
Clients stream live packet telemetry over HTTP/2 using Server-Sent Events. The stream supports server-side filtering by interface, protocol, IP address, and port:
# Stream live TCP packets on interface eth0 with SYN or RST flags curl -N "https://192.168.1.1:2966/api/v1/events/packets?iface=eth0&proto=tcp"
Example Event Payload (JSON)
{
"timestamp": "2026-10-13T07:15:22.408Z",
"interface": "wan0",
"direction": "in",
"ethernet": {
"src": "00:1a:2b:3c:4d:5e",
"dst": "52:54:00:12:34:56",
"type": "IPv4"
},
"ip": {
"version": 4,
"src": "198.51.100.24",
"dst": "192.168.1.1",
"ttl": 54,
"proto": "TCP",
"length": 60
},
"tcp": {
"srcPort": 52414,
"dstPort": 443,
"seq": 284192041,
"flags": ["SYN"]
},
"action": "pass"
}9. VictoriaMetrics, ClickHouse & Insight Flow Analytics
In addition to real-time eBPF and NFLOG streaming, historical time-series metrics are stored in an embedded, resource-efficient VictoriaMetrics engine and optional ClickHouse columnar store. This replaces legacy RRDtool graphs with high-speed drill-down analytics for bandwidth usage, top talkers, K3s pod flows, and application protocols.
Reporting > eBPF & K3s Observability (/reporting/ebpf), Reporting > Live Flow Monitor (/reporting/flows), and Diagnostics > Packet Tracer (/diagnostics/packet-tracer).