the long story short - they are stored in array effective and in list progs in cgroup->bpfwith what cgroup roots actually located on this machine:
can you find in /proc/cgroups roots with hierarchy ID 0 and 1?
Each eBPF program stored in struct bpf_prog. But wait - we can have several eBPF programs connected to the same event, right? Yeah, so for such cases there is another struct - bpf_prog_array. kernel devs were so generous that they even gave us function bpf_prog_array_length. Unfortunately it is not exported
What can this useful information give us? Well, there is function cgroup_bpf_prog_detach. And suddenly your eBPF program may stop receiving messages
Below I will try to explain boring and dirty details
cgroups
This article says:
hierarchy: a set of cgroups arranged in a treeso we need to find roots and then just traverse this trees. Roots have type cgroup_root and stored in cgroup_hierarchy_idr (synced with mutex cgroup_mutex). As usually linux lies - lets compare content of /proc/cgroups:
#subsys_name hierarchynum_cgroups enabledcpuset611cpu511cpuacct511blkio411memory21481devices9991freezer1011net_cls711perf_event811net_prio711hugetlb311pids111031rdma1211
[0] at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144[1] systemd at 0xffff8fd6816ea000 flags 4 hierarchy_id 1 nr_cgrps 145 real_cnt 144[2] at 0xffff8fd68297a000 flags 0 hierarchy_id 2 nr_cgrps 148 real_cnt 147[3] at 0xffff8fd68297c000 flags 0 hierarchy_id 3 nr_cgrps 1 real_cnt 0[4] at 0xffff8fd682978000 flags 0 hierarchy_id 4 nr_cgrps 1 real_cnt 0[5] at 0xffff8fd68297e000 flags 0 hierarchy_id 5 nr_cgrps 1 real_cnt 0[6] at 0xffff8fd6854c8000 flags 0 hierarchy_id 6 nr_cgrps 1 real_cnt 0[7] at 0xffff8fd6854ce000 flags 0 hierarchy_id 7 nr_cgrps 1 real_cnt 0[8] at 0xffff8fd6854ca000 flags 0 hierarchy_id 8 nr_cgrps 1 real_cnt 0[9] at 0xffff8fd6854cc000 flags 0 hierarchy_id 9 nr_cgrps 99 real_cnt 98[10] at 0xffff8fd685e16000 flags 0 hierarchy_id 10 nr_cgrps 1 real_cnt 0[11] at 0xffff8fd685e12000 flags 0 hierarchy_id 11 nr_cgrps 103 real_cnt 102[12] at 0xffff8fd685e14000 flags 0 hierarchy_id 12 nr_cgrps 1 real_cnt 0can you find in /proc/cgroups roots with hierarchy ID 0 and 1?
How to traverse this tree? It starts in field cgrp->self and we can use functions css_next_descendant_pre/css_next_descendant_post etc. Strictly speaking they return pointer to cgroup_subsys_state but this is first field self in cgroup so casting is safe
eBPF
eBPF programs are stored in prog_idr (synced with spinlock_t prog_idr_lock). Lets see what we have:
sudo ./lkmem -d -c -B ~/krnl/curr ~/krnl/System.map-5.11.0-40-genericprog_idr at 0xffffffff8e9be540: 23 [0] prog 0xffff9f8e809a7000 id 31 len 123 jited_len 555 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05c2254 [1] prog 0xffff9f8e809bf000 id 32 len 1824 jited_len 8195 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05ba9e0 [2] prog 0xffff9f8e80905000 id 33 len 1343 jited_len 6186 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc00612b0 [3] prog 0xffff9f8e809c7000 id 34 len 1682 jited_len 7822 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc02d6040 [4] prog 0xffff9f8e809a9000 id 35 len 1209 jited_len 5510 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05be370 [5] prog 0xffff9f8e809cf000 id 36 len 1397 jited_len 6396 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05c46d8 [6] prog 0xffff9f8e809d7000 id 37 len 1223 jited_len 5578 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05c7108 [7] prog 0xffff9f8e80055000 id 38 len 267 jited_len 1237 type: 5 BPF_PROG_TYPE_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc0059254 [8] prog 0xffff9f8e8005d000 id 39 len 247 jited_len 1116 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05ca90c [9] prog 0xffff9f8e80115000 id 40 len 217 jited_len 994 type: 5 BPF_PROG_TYPE_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05cc2c0 [10] prog 0xffff9f8e8011d000 id 41 len 744 jited_len 3405 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05ce098 [11] prog 0xffff9f8e809df000 id 42 len 633 jited_len 2701 type: 5 BPF_PROG_TYPE_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05d0234 [12] prog 0xffff9f8e808f4000 id 43 len 492 jited_len 2233 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05d262c [13] prog 0xffff9f8e809bb000 id 44 len 68 jited_len 312 type: 17 BPF_PROG_TYPE_RAW_TRACEPOINT expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05d41f4 [14] prog 0xffff9f8e809f1000 id 55 len 2 jited_len 15 type: 1 BPF_PROG_TYPE_SOCKET_FILTER expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05e49f4 [15] prog 0xffff9f8e8003b000 id 89 len 8 jited_len 54 type: 8 BPF_PROG_TYPE_CGROUP_SKB expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc04fab24 [16] prog 0xffff9f8e80037000 id 90 len 8 jited_len 54 type: 8 BPF_PROG_TYPE_CGROUP_SKB expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc0561468 [17] prog 0xffff9f8e80048000 id 91 len 8 jited_len 54 type: 8 BPF_PROG_TYPE_CGROUP_SKB expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc0563828 [18] prog 0xffff9f8e8004f000 id 92 len 8 jited_len 54 type: 8 BPF_PROG_TYPE_CGROUP_SKB expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc0565594 [19] prog 0xffff9f8e80051000 id 93 len 8 jited_len 54 type: 8 BPF_PROG_TYPE_CGROUP_SKB expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc056747c [20] prog 0xffff9f8e80053000 id 94 len 8 jited_len 54 type: 8 BPF_PROG_TYPE_CGROUP_SKB expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc056923c [21] prog 0xffff9f8e8012f000 id 95 len 8 jited_len 54 type: 8 BPF_PROG_TYPE_CGROUP_SKB expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc058e308 [22] prog 0xffff9f8e80131000 id 96 len 8 jited_len 54 type: 8 BPF_PROG_TYPE_CGROUP_SKB expected_attach_type: 0 BPF_CGROUP_INET_INGRESS bpf_func: 0xffffffffc05900dc
putting it all together
Now we know how to
- find and traverse cgroups
- where eBPF stored in each cgroup
- can enum ePBF programs from bpf_prog_array
Lets see what we have:
sudo ./lkmem -d -c -g ~/krnl/curr ~/krnl/System.map-5.11.0-40-generic[0] at 0xffffffff8e9a2200 flags 8 hierarchy_id 0 nr_cgrps 145 real_cnt 144 child 9: cgroup at 0xffff8fd6824aa000 serial_nr 80 flags 0 level 2 cgroup BPF: BPF_CGROUP_INET_INGRESS: 0xffff8fd68d361300 cnt 1 flags 0 [0] prog 0xffff9f8e80037000 id 90 type 8 len 8 jited_len 54 bpf_func: 0xffffffffc0561468 BPF_CGROUP_INET_EGRESS: 0xffff8fd687cf5f00 cnt 1 flags 0 [0] prog 0xffff9f8e8003b000 id 89 type 8 len 8 jited_len 54 bpf_func: 0xffffffffc04fab24 child 24: cgroup at 0xffff8fd68864b000 serial_nr 286 flags 0 level 2 cgroup BPF: BPF_CGROUP_INET_INGRESS: 0xffff8fd6858fea40 cnt 1 flags 0 [0] prog 0xffff9f8e80053000 id 94 type 8 len 8 jited_len 54 bpf_func: 0xffffffffc056923c BPF_CGROUP_INET_EGRESS: 0xffff8fd5d1fb6fc0 cnt 1 flags 0 [0] prog 0xffff9f8e80051000 id 93 type 8 len 8 jited_len 54 bpf_func: 0xffffffffc056747c child 44: cgroup at 0xffff8fd68310c000 serial_nr 596 flags 0 level 2 BPF_CGROUP_INET_INGRESS: 0xffff8fd5c5be97c0 cnt 1 flags 0 [0] prog 0xffff9f8e80131000 id 96 type 8 len 8 jited_len 54 bpf_func: 0xffffffffc05900dc BPF_CGROUP_INET_EGRESS: 0xffff8fd687cf7300 cnt 1 flags 0 [0] prog 0xffff9f8e8012f000 id 95 type 8 len 8 jited_len 54 bpf_func: 0xffffffffc058e308 child 94: cgroup at 0xffff8fd5834d7000 serial_nr 1136 flags 0 level 2 cgroup BPF: BPF_CGROUP_INET_INGRESS: 0xffff8fd5c7cd1f00 cnt 1 flags 0 [0] prog 0xffff9f8e8004f000 id 92 type 8 len 8 jited_len 54 bpf_func: 0xffffffffc0565594 BPF_CGROUP_INET_EGRESS: 0xffff8fd68555fac0 cnt 1 flags 0 [0] prog 0xffff9f8e80048000 id 91 type 8 len 8 jited_len 54 bpf_func: 0xffffffffc0563828
What can this useful information give us? Well, there is function cgroup_bpf_prog_detach. And suddenly your eBPF program may stop receiving messages
Link to source code