$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill ebpf-observabilityUse eBPF for deep kernel-level observability — trace syscalls, network flows, and application behavior without code changes using Cilium, Tetragon, and bpftrace.
| 1 | # eBPF Observability |
| 2 | |
| 3 | eBPF (extended Berkeley Packet Filter) allows you to run sandboxed programs in the Linux kernel without modifying kernel source code or loading kernel modules. This skill covers using eBPF for deep observability, network monitoring, and security enforcement across cloud-native infrastructure. |
| 4 | |
| 5 | --- |
| 6 | |
| 7 | ## 1. When to Use |
| 8 | |
| 9 | Use eBPF-based observability when you need: |
| 10 | |
| 11 | - **Deep performance debugging** -- trace kernel-level latency, syscall overhead, and scheduling delays that application-level metrics cannot reveal. |
| 12 | - **Network observability without sidecars** -- capture L3/L4/L7 flows, DNS queries, and TCP state transitions directly from the kernel, eliminating the CPU and memory overhead of sidecar proxies. |
| 13 | - **Security monitoring at the kernel boundary** -- detect container escapes, unexpected process execution, sensitive file access, and anomalous syscall patterns in real time. |
| 14 | - **Continuous profiling in production** -- generate CPU flame graphs and memory allocation profiles with negligible overhead (typically under 1% CPU). |
| 15 | - **Service mesh replacement or augmentation** -- Cilium can replace kube-proxy and provide identity-aware network policies enforced at the kernel level. |
| 16 | |
| 17 | Avoid eBPF when your kernel version is below 4.19, when you are running on managed platforms that restrict BPF capabilities, or when your debugging needs are fully met by application-level tracing. |
| 18 | |
| 19 | --- |
| 20 | |
| 21 | ## 2. Prerequisites |
| 22 | |
| 23 | ### Kernel Version Requirements |
| 24 | |
| 25 | | Feature | Minimum Kernel | Recommended Kernel | |
| 26 | |--------------------------|----------------|--------------------| |
| 27 | | Basic BPF maps & probes | 4.9 | 5.10+ | |
| 28 | | BPF CO-RE (BTF support) | 5.2 | 5.10+ | |
| 29 | | BPF ring buffer | 5.8 | 5.10+ | |
| 30 | | BPF LSM hooks | 5.7 | 5.15+ | |
| 31 | | Cilium full features | 4.19 | 5.10+ | |
| 32 | | Tetragon | 4.19 | 5.13+ | |
| 33 | |
| 34 | ### Verify Kernel Support |
| 35 | |
| 36 | ```bash |
| 37 | # Check kernel version |
| 38 | uname -r |
| 39 | |
| 40 | # Verify BTF (BPF Type Format) is enabled -- required for CO-RE |
| 41 | ls /sys/kernel/btf/vmlinux |
| 42 | |
| 43 | # Check BPF filesystem is mounted |
| 44 | mount | grep bpf |
| 45 | |
| 46 | # If not mounted, mount it |
| 47 | sudo mount -t bpf bpf /sys/fs/bpf |
| 48 | |
| 49 | # Verify BPF JIT is enabled |
| 50 | cat /proc/sys/net/core/bpf_jit_enable |
| 51 | # Should return 1; if not: |
| 52 | sudo sysctl net.core.bpf_jit_enable=1 |
| 53 | ``` |
| 54 | |
| 55 | ### Install Toolchain |
| 56 | |
| 57 | ```bash |
| 58 | # Ubuntu/Debian -- install bpftrace, bcc tools, and libbpf |
| 59 | sudo apt-get update |
| 60 | sudo apt-get install -y bpftrace bpfcc-tools libbpf-dev linux-headers-$(uname -r) |
| 61 | |
| 62 | # Fedora/RHEL |
| 63 | sudo dnf install -y bpftrace bcc-tools libbpf-devel kernel-devel |
| 64 | |
| 65 | # Verify bpftrace works |
| 66 | sudo bpftrace -e 'BEGIN { printf("eBPF is working\n"); exit(); }' |
| 67 | ``` |
| 68 | |
| 69 | --- |
| 70 | |
| 71 | ## 3. Cilium Setup |
| 72 | |
| 73 | Cilium replaces kube-proxy with eBPF-based networking, providing identity-aware security and deep network observability via Hubble. |
| 74 | |
| 75 | ### Install Cilium on Kubernetes |
| 76 | |
| 77 | ```bash |
| 78 | # Add the Cilium Helm repo |
| 79 | helm repo add cilium https://helm.cilium.io/ |
| 80 | helm repo update |
| 81 | |
| 82 | # Install Cilium with Hubble enabled |
| 83 | helm install cilium cilium/cilium --version 1.16.4 \ |
| 84 | --namespace kube-system \ |
| 85 | --set kubeProxyReplacement=true \ |
| 86 | --set k8sServiceHost="${API_SERVER_IP}" \ |
| 87 | --set k8sServicePort="${API_SERVER_PORT}" \ |
| 88 | --set hubble.enabled=true \ |
| 89 | --set hubble.relay.enabled=true \ |
| 90 | --set hubble.ui.enabled=true \ |
| 91 | --set hubble.metrics.enableOpenMetrics=true \ |
| 92 | --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload}" |
| 93 | |
| 94 | # Wait for Cilium to be ready |
| 95 | cilium status --wait |
| 96 | ``` |
| 97 | |
| 98 | ### Install the Cilium CLI and Hubble CLI |
| 99 | |
| 100 | ```bash |
| 101 | # Cilium CLI |
| 102 | CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) |
| 103 | curl -L --remote-name "https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" |
| 104 | sudo tar xzvf cilium-linux-amd64.tar.gz -C /usr/local/bin |
| 105 | rm cilium-linux-amd64.tar.gz |
| 106 | |
| 107 | # Hubble CLI |
| 108 | HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt) |
| 109 | curl -L --remote-name "https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz" |
| 110 | sudo tar xzvf hubble-linux-amd64.tar.gz -C /usr/local/bin |
| 111 | rm hubble-linux-amd64.tar.gz |
| 112 | ``` |
| 113 | |
| 114 | ### Hubble Network Observability |
| 115 | |
| 116 | ```bash |
| 117 | # Port-forward the Hubble Relay |
| 118 | cilium hubble port-forward & |
| 119 | |
| 120 | # Observe all flows in real time |
| 121 | hubble observe --follow |
| 122 | |
| 123 | # Filter flows by namespace |
| 124 | hubble observe --namespace production --follow |
| 125 | |
| 126 | # Filter by verdict (dropped traffic) |
| 127 | hubble observ |