Continuing our series of articles on BPF - the general-purpose virtual machine of the Linux kernel - in this issue we will talk about what types of BPF programs exist and how they are used in the real world of capitalist cash. In addition, at the end of the article there are a number of links, in particular, to two and a half existing books about BPF.

Linux kernel 5.9 defines more than thirty different types of BPF programs, and I am going to write several articles about some of these types, so this article, inevitably, is an overview and does not contain as many technical details as the previous ones. But, nevertheless, we will try to finally answer the question why all this is needed and why there is so much noise around BPF.

If you want to know exactly how BPF helps to effectively solve the problems of protection against DDoS attacks, load balancing on servers, implementation of the kubernetes network stack, protection of systems from attack, effective tracing of 24x7 systems right in production, and many others, then welcome under cat.

Program types and table of contents

All existing BPF program types are registered in include/uapi/linux/bpf.hthe Linux kernel file . In the following sections, I tried to group them into logical groups (asterisks mark the technical educational program subsections):

• Linux microkernel
• BPF tracing and data collection programs
• Linux Security Modules
• Dynamically Extending BPF Programs
• Linux Infrared Remote Control
• BPF Network Programming Programs
  
  
  • * Linux networking stack: packet arrival
  • Steroid Networks - Express Data Path
  • * The struct __sk_buff structure
  • Linux Traffic Control
  • BPF
  • « » flower
  • BPF
  • BPF
  • LWT:

BPF .

, - , , BPF_PROG_* .

Linux

1992 «» (- ) ( ). , «LINUX is obsolete» , Linux ( 1992 ) , . «» , , , :

«, linux , , . , , , . ( ) linux . GNU , : , , . Linux , GNU " "»

, — BPF Linux . 2020 Martin KaFai Lau , . —  - , , - .

BPF: BPF_PROG_TYPE_STRUCT_OPS. , Daniel Borkman , BPF — , .

, BPF. - , . BPF tcp_congestion_ops, TCP congestion control. —  DCTCP CUBIC BPF.

, , BPF, , , (, BPF ) . , , — . . BPF Summit.

BPF

BPF Brendan Gregg, , Linux . bcc, bpftrace, «BPF Performance Tools», , BPF, .. Facebook Netflix, , BPF, 24x7. BPF — BPF .

? . BPF :

• () Linux
• tracepoint
• perf, software hardware

maps, , . , BPF, , , .

( bpftrace, ):

#! /usr/bin/env bpftrace

#include <linux/skbuff.h>
#include <linux/ip.h>

k:icmp_echo {
    $skb = (struct sk_buff *) arg0;
    $iphdr = (struct iphdr *) ($skb->head + $skb->network_header);
    @pingstats[ntop($iphdr->saddr), ntop($iphdr->daddr)]++;
}

, . kprobe icmp_echo, ICMPv4 echo request. , arg0 , — sk_buff, . IP @pingstats. , , IP ! , kprobe, user space, .

BPF, tracing:

• BPF_PROG_TYPE_KPROBE: BPF kprobe, kretprobe, uprobe uretprobe. , (.. ), , , .
• BPF_PROG_TYPE_PERF_EVENT: BPF perf.
• BPF_PROG_TYPE_TRACEPOINT: BPF tracepoint. , kprobes? , tracepoints — API ( , / tracepoint ) , tracepoints ( ).
• BPF_PROG_TYPE_RAW_TRACEPOINT: tracepoints . raw tracepoints BPF «» , ,
• BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE: tracepoints (. )
• BPF_PROG_TYPE_TRACING: , : tracepoints, , , ( : sudo cat /sys/kernel/debug/error_injection/list), «». , BTF — .

, —  Linux BPF, .

Linux Security Modules

Linux (security hooks), , , , .. Linux (Linux Security Modules LSM) SELinux, AppArmor, .., .

, . , API Kernel Runtime Security Instrumentation LSS-NA 2019. BPF, BPF_PROG_TYPE_LSM, , BPF LSM . , , BPF, ..

, , BPF, . , BPF. , LSM . user mode helper, BPF libbpf, libbpf .

KRSI KP Singh, KRSI, BPF Summit.

BPF

Tail calls

, , BPF . , BPF 4096 . , BPF . —  . tail calls.

tail calls . , , —  . BPF_MAP_TYPE_PROG_ARRAY, BPF ( ):

- bpf_tail_call. , , bpf_tail_call(&map, ctx, 1), ctx —  , . , long jump, . 32, , .

, 5.1 , , . tail calls, .

, , tail calls . , bpf_tail_call, ? — , .

tail calls . BPF, BPF_PROG_TYPE_EXT, . BPF trampoline TRACING , —  , .

, , xdp-dispatcher — , XDP . «» -, XDP , - . . Multiple XDP programs on a single interface—status and next steps Toke Høiland-Jørgensen Linux Plumbers 2020.

LIRC: Linux Infrared Remote Control

, BPF BPF_PROG_TYPE_LIRC_MODE2 . lwn Sean Young, , , .

, BPF , / . BPF , , , map. , - , -. , , bpf_rc_keydown :

, , lirc? Sean Young , BPF , , API: IR userspace ( ).

BPF

BPF Berkeley Labs, BPF Linux, , BPF .

«» —  XDP Linux —  - , / Linux.

Linux:

, BPF , , XDP, Linux. . ( , XDP, , .)

, Linux . , DMA , CPU. , , CPU, RAM.

Linux — top half bottom half. — , , (top half), (bottom half), softirq . , bottom halves, , softirq NET_RX.

softirq , struct sk_buff. sk_buff, socket buffer, —  Linux. Linux sk_buff. , : , , -, .., .. , sk_buff .

. , head end , data tail , net_header transport_hdr , , .. data —  «» .

netif_receive_skb . ? Netfilter wiki:

sk_buff , (ingress qdisc ), , netfiler. , (sk_buff) , — .

, ( ). start, softirq, eBPF XDP , ...

— Express Data Path

sk_buff — , , VLAN .. BPF XDP (Express Data Path) sk_buff.

XDP , , , RAM . struct xdp_md, , , . — —  XDP (XDP_DROP), , (XDP_TX), (XDP_REDIRECT), (XDP_PASS):

, / , , , , , MAC , XDP , / .

XDP AF_XDP, , , zero copy. , DPDK, :

AF_XDP: AF_XDP (rx queue), . XDP, . , , , . ( , , , . , UDP 65784 AF_XDP, 13, , , : ethtool -N flow-type udp4 dst-port 65784 action 13.)

, XDP , . , , CPU 0%. Netronome, , .

«» XDP, : DDoS . , Facebook, load balancer katran, XDP, Cloudfare XDP DDoS load balancing, cilium XDP , .. R&D , XDP P4 , , ( NPU — Networking Processing Unit).

XDP , — , . , XDP, XDP Tutorial, , , — kozlyuk .

struct __sk_buff

BPF, . , Linux sk_buff, . len — , network_header — L3, dev — struct net_device , .

, — sk_buff ( XDP, sk_buff ), , BPF sk_buff. , —  BPF struct __sk_buff:

struct __sk_buff {
    __u32 len;
    __u32 pkt_type;
    __u32 mark;
    __u32 queue_mapping;
    __u32 protocol;
    __u32 vlan_present;
    ...
};

__sk_buff sk_buff, Verifier , . BPF __sk_buff :

int bpf_prog(struct __sk_buf *ctx)
{
    __u32 len = ctx->len;
    __u32 type = ctx->pkt_type;
    ...
}

Verifier ( ) , sk_buff. , :

, Verifier . , pkt_type, 3, Verifier , .

#include <linux/bpf.h>

__attribute__((section("socket/test")))
int bpf_prog(struct __sk_buff *ctx)
{
    __u32 len = ctx->len;
    __u32 type = ctx->pkt_type;
    return len + type;
}

clang -target bpf -O2 skbuff.c -o skbuff.o -c

mkdir mnt
sudo mount -t bpf none ./mnt
bpftool prog load ./skbuff.o ./mnt/xxx

$ llvm-objdump -D ./skbuff.o --section socket/test
  0:    61 12 00 00 00 00 00 00 r2 = *(u32 *)(r1 + 0)
  1:    61 10 04 00 00 00 00 00 r0 = *(u32 *)(r1 + 4)
  2:    0f 20 00 00 00 00 00 00 r0 += r2
  3:    95 00 00 00 00 00 00 00 exit

$ sudo bpftool prog dump xlated pinned ./mnt/xxx
   0: (61) r2 = *(u32 *)(r1 +104)
   1: (71) r0 = *(u8 *)(r1 +120)
   2: (54) w0 &= 7
   3: (0f) r0 += r2
   4: (95) exit

Linux

Linux, , , sk_buff, ingress qdisc. , , , egress qdisc — , , / netfilter.

Qdisc queueing discipline Linux — Traffic Control (TC). egress qdisc —  , . , - , .

— classful classless —  . — , . , egress qdisc, pfifo_fast, TOS IPv4 IPv6 ( . lartc 9.2):

— qdisc noqueue, , , .

Classful qdiscs . qdiscs. , . . , , : (classifiers) (actions). , , , , . : u32, flower .. : drop ( ), reclassify ( , , , VLAN tag), ..

, qdiscs , . qdiscs , C ? BPF, BPF_PROG_TYPE_SCHED_CLS BPF_PROG_TYPE_SCHED_ACT, , . , qdisc clsact, egress, ingress, BPF BPF_PROG_TYPE_SCHED_CLS direct action. — BPF — actions, .. .

BPF TC, — BPF Reference Guide Daniel Borkman cilium —  CNI kubernetes, Alibaba Google.

BPF

BPF — BPF , . eBPF cBPF, , eBPF cBPF. , BPF BPF_PROG_TYPE_SOCKET_FILTER SO_ATTACH_BPF. , , CAP_SYS_ADMIN.

BPF BPF_PROG_TYPE_SOCKET_FILTER , :

• , BPF, BPF, , ( sk_buff) : , (, ). , RAW , . BPF_PROG_TYPE_SOCKET_FILTER Evil eBPF In-Depth DEFCONF 27.
• AF_PACKET PACKET_FANOUT, . , , DPI. Linux . 2015 fanout PACKET_FANOUT_DATA, BPF.
• 2007 xt_bpf netfilter. BPF, . 2016 eBPF — eBPF BPF_PROG_TYPE_SOCKET_FILTER.
• , , , tun , , , , VM. . TUNSETFILTEREBPF.
• tun BPF . . TUNSETSTEERINGEBPF.
• Kernel Connection Multiplexor TCP datagram (. lwn, kcm). TCP BPF BPF_PROG_TYPE_SOCKET_FILTER, AF_KCM SIOCKCMATTACH (. , ).
• , BPF_PROG_TYPE_SOCKET_FILTER SO_ATTACH_REUSEPORT_EBPF, . Perfect locality and three epic SystemTap scripts.

« » flower

__skb_flow_dissect, , Linux flow dissector — - . , , ingress Linux, flower.

, , ,   . BPF — BPF_PROG_TYPE_FLOW_DISSECTOR, BPF. namespace.

BPF

(cgroups) . , BPF : BPF cgroup. , ( , ) -, . cgroups , , , , - . BPF. , .

BPF_PROG_TYPE_CGROUP_SKB BPF (ingress) (egress) . 1, , 0, . , . , , .. : BPF systemd.

, BPF . , BPF BPF_PROG_TYPE_CGROUP_SOCK , struct sock. sk_bound_dev_if , . bind(2) / .

, , BPF BPF_PROG_TYPE_CGROUP_SOCK_ADDR. bind IP , ( use case : cgroup , , . ). connect, getpeername, getsockname, sendmsg recvmsg. , , , cilium iptables k8s.

BPF_PROG_TYPE_CGROUP_SOCKOPT setsockopt.

BPF_PROG_TYPE_CGROUP_DEVICE cgroupv2 , device cgroupsv1.

BPF_PROG_TYPE_CGROUP_SYSCTL sysctl , , , cgroup .

.

BPF

BPF_PROG_TYPE_SK_SKB . : SOCKMAP, . , , recvmsg, BPF, sk_buff . , Isovalent CNI cilium k8s, Cloudfare, . SOCKMAP — TCP splicing of the future.

BPF_PROG_TYPE_SK_SKB, BPF_PROG_TYPE_SK_MSG , sendmsg sendpage , L7 — , . BPF_PROG_TYPE_SK_SKB, sockmap .

BPF_PROG_TYPE_SK_REUSEPORT , SO_REUSEPORT. BPF, , .

BPF_PROG_TYPE_SK_LOOKUP , . : , IP , , , . namespaces.

, , TCP — BPF_PROG_TYPE_SOCK_OPS. cgroupv2, BPF_PROG_TYPE_CGROUP_SOCKOPT, , .., . , TCP , .

LWT:

, , . . , IPv4- IPv6-, VPN, .

, , . , , , , .

, Linux, : ip link add name ipip0 type ipip... .. 2015 Linux . , , , —  .

, , 2016 , ! ,   BPF :

• BPF_PROG_TYPE_LWT_IN: lwtunnel_input
• BPF_PROG_TYPE_LWT_OUT: lwtunnel_output
• BPF_PROG_TYPE_LWT_XMIT: lwtunnel_xmit

input , output — , xmit —  . struct __sk_buff, (BPF_OK), (BPF_DROP), (BPF_REDIRECT) , , (BPF_DROP). , xmit —  , .

netlink . - , BPF iproute2, :

ip route add 10.0.0.0/24 encap bpf xmit obj <prog.o> section <section> dev <dev>

<prog.o> — BPF ELF, <section> —  .

2018 , BPF_PROG_TYPE_LWT_SEG6LOCAL, seg6local, . Using SRv6.

BPF: BPF_PROG_TYPE_UNSPEC. , , / . bpf(2) .

, ! 99% , . , BPF Linux BPF_PROG_TYPE_UNSPEC BPF, , , , tcpdump wireshark Linux, .

,

BPF Linux, , - . BPF , , Linux. , BPF Linux.

(, , ) Linux — BPF kprobes, tracepoints perf events, — libbpf, bcc bpftrace.

1. BPF , : classic BPF
2. BPF , : extended BPF

2,5

• Brendan Gregg, «BPF Performance Tools». BPF Linux —  BCC, . BPF .
• Brendan Gregg, «Systems Performance: Enterprise and the Cloud, 2nd Edition (2020)». «Systems Performance». : BPF, Solaris, . «BPF Performance Tools» «?», «?»
• David Calavera and Lorenzo Fontana, «Linux Observability with BPF». . BPF, , , .

Online-,

There are a great many articles and reports about BPF. Therefore, we will take advantage of the fact that the above-mentioned Isovalent company is trying to lead in the HYIP harvest with BPF and, in particular, recently created this site with documentation and held a BPF summit - a mini-conference about BPF. Fun fact: the participants of the aforementioned BPF Summit chose a new BPF mascot, the bee, and came up with the euphonious name Ebee: