Saturday, October 3, 2020

XDP - Getting Started with XDP (Linux)

 

XDP


Introduced in Linux 4.8 eBPF hook at the driver level (ingress)

Intercept packet before it reaches the stack, before allocating sk_buff

Rationale: implement a faster data path which is part of the kernel, maintained by the kernel community Rather for simple use cases.

Complex processing: forward to stack Not a “kernel bypass”, works in cooperation with the networking stack


Essentially, user-space networking achieves high-speed performance by moving packet-processing out of the kernel’s realm into user-space

XDP does in fact the opposite: it moves user-space networking programs (filters, mappers, routing, etc) into the kernel’s realm. 

XDP allows us to execute our network function as soon as a packet hits the NIC, and before it starts moving upwards into the 

kernel’s networking layer, which results into a significant increase of packet-processing speed

Accelerating-VM-Networking-through-XDP_Jason-Wang.pdf

https://help.netronome.com/support/solutions/articles/36000050009-agilio-ebpf-2-0-6-extended-berkeley-packet-filter

https://www.netronome.com/blog/hello-xdp_drop/

https://archive.fosdem.org/2018/schedule/event/xdp/attachments/slides/2220/export/events/attachments/xdp/slides/2220/fosdem18_SdN_NFV_qmonnet_XDPoffload.pdf


XDP MODES

In total, XDP supports three operation modes which iproute2 implements as well: xdpdrv, xdpoffload and xdpgeneric.

xdpdrv stands for native XDP, meaning the BPF program is run directly in the driver’s receive path at the earliest possible point in software.
This is the normal / conventional XDP mode and requires driver’s to implement XDP support, which all major 10G/40G/+ networking drivers
in the upstream Linux kernel already provide.

xdpgeneric stands for generic XDP and is intended as an experimental test bed for drivers which do not yet support native XDP.
Given the generic XDP hook in the ingress path comes at a much later point in time when the packet already enters the stack’s
main receive path as a skb, the performance is significantly less than with processing in xdpdrv mode.
xdpgeneric therefore is for the most part only interesting for experimenting, less for production environments.

xdpoffload Last but not least, thIs mode is implemented by SmartNICs such as those supported by Netronome’s nfp driver and
allow for offloading the entire BPF/XDP program into hardware, thus the program is run on each packet reception directly
on the card. This provides even higher performance than running in native XDP although not all BPF map types or BPF helper
functions are available for use compared to native XDP. The BPF verifier will reject the program in such case and report
to the user what is unsupported. Other than staying in the realm of supported BPF features and helper functions,
no special precautions have to be taken when writing BPF C programs.











#include <linux/bpf.h>
int main()
{
return XDP_DROP;
}


clang -target bpf -O2 -c xdp.c -o xdp.o


ip -force link set dev ens1f0 xdpdrv obj xdp.o sec .text


ip link show ens1f0
32: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether f4:e9:d4:ed:25:38 brd ff:ff:ff:ff:ff:ff
prog/xdp id 36


0 XDP_ABORTED - Error, Block the packet
1 XDP_DROP - Block the packet
2 XDP_PASS - Allow the packet to continue up the kernel
3 XDP_TX - Bounce the packet back in the direction it came from



$ hping3 [IP Address of Host]
Traffic can be monitored using tcpdump, however it will show that no packets are received.
This is due to XDP dropping packets at the start of the kernel path, before the packets can reach tcpdump

unload xdp
ip link set dev [DEV] xdpdrv off


H/w offload load
ip -force link set dev ens1f0 xdpoffload obj xdp.o sec .text


Testing XDP



Steps 

Description  


Step 1

Check "clang" is installed or not else install it by yum install clang and XDP only supports on RHEL8 and above kernel 


Step 2

Create xdp_drop.c file in "/usr/src/kernels/$(uname -r)/net/xdp" directory 

touch /usr/src/kernels/$(uname -r)/net/xdp/xdp_drop.c


Step 3 

Write xdp_drop code inside xdp_drop.c file



#include <linux/bpf.h>

 #ifndef __section

 # define __section(NAME)                  \

    __attribute__((section(NAME), used))

 #endif

 

 __section("prog")

 int xdp_drop(struct xdp_md *ctx)

 {

     return XDP_DROP;

 }

 char __license[] __section("license") = "GPL";




Step 4 

Compile this code with below command so that it will create obj file

clang -O2 -Wall -target bpf -c xdp_drop.c -o xdp_drop.o


Step 5

Insert/Probe xdp_drop.o file to both interface (PF) with below command

ip link set dev ens3f0 xdp obj xdp_drop.o

ip link set dev ens3f1 xdp obj xdp_drop.o



Step 6

With  "ip link show"  command  check xdp loaded with some id.


[root@Gen9-XDP-Host-RHEL8 xdp]# ip link show

4: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq state UP mode DEFAULT group default qlen 1000

     link/ether 00:0e:1e:d6:62:fc brd ff:ff:ff:ff:ff:ff

     prog/xdp id 1 tag f95672269956c10d jited

 5: ens3f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq state UP mode DEFAULT group default qlen 1000

     link/ether 00:0e:1e:d6:62:fd brd ff:ff:ff:ff:ff:ff

     prog/xdp id 2 tag f95672269956c10d jited


Step 7

Send Traffic through scapy tool from Peer system to both the interface simultaneously 



sendp (Ether(src="00:0e:1e:d6:62:fc",dst="14:02:ec:d3:af:0a")/IP(src="44.44.44.1",dst="55.55.55.1")/TCP(sport=0xbbbb,dport=0xaaaa)/("x"*200), iface="ens3f0",count=1000000)

sendp (Ether(src="00:0e:1e:d6:62:fd",dst="14:02:ec:d3:af:0b")/IP(src="44.44.44.1",dst="55.55.55.1")/TCP(sport=0xbbbb,dport=0xaaaa)/("x"*200), iface="ens3f1",count=1000000)


1. Observed that packets were being dropped and “xdp_no_pass” counters were increasing, No packets were seen in tcpdump that suggest that Xpress data path was being used


[root@Gen9-XDP-Host-RHEL8 xdp]# ethtool -S ens3f0 | grep xdp

      0: xdp_no_pass: 5000

      1: xdp_no_pass: 3731

      2: xdp_no_pass: 5000

      3: xdp_no_pass: 4000

      4: xdp_no_pass: 4609

      5: xdp_no_pass: 5000

      6: xdp_no_pass: 4000

      7: xdp_no_pass: 5000


2. You should not see any unexpected failures in dmesg or /var/log/messages

 3. Should not see any driver/FW failure messages or system hang. 




Loading IN NATIVE MODE

# ip -force link set dev em1 xdpdrv obj prog.o
# ip link show
[...]
6: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp qdisc mq state UP mode DORMANT group default qlen 1000
link/ether be:08:4d:b6:85:65 brd ff:ff:ff:ff:ff:ff
prog/xdp id 1 tag 57cd311f2e27366b
[...]
# ip link set dev em1 xdpdrv off

The option verb can be appended for loading programs in order to dump the verifier log,
# ip -force link set dev em1 xdpdrv obj prog.o verb

LOADING IN GENERIC MODE

# ip -force link set dev em1 xdpgeneric obj prog.o
# ip link show
[...]
6: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc mq state UP mode DORMANT group default qlen 1000
link/ether be:08:4d:b6:85:65 brd ff:ff:ff:ff:ff:ff
prog/xdp id 4 tag 57cd311f2e27366b <-- BPF program ID 4
[...]
# bpftool prog dump xlated id 4 <-- Dump of instructions running on em1
0: (b7) r0 = 1
1: (95) exit
# ip link set dev em1 xdpgeneric off


XDP Related Config
==================
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_CLS_ACT=y
CONFIG_BPF_JIT=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_TEST_BPF=m
CONFIG_XDP_SOCKETS=y

$ cd tools/testing/selftests/bpf/
$ make
$ sudo ./test_verifier


Sample COde to drop IP Traffic from 50.50.50.1

#include "../../include/uapi/linux/bpf.h"

#include "../../include/uapi/linux/if_ether.h"

#include "../../include/uapi/linux/if_packet.h"

#include "../../include/uapi/linux/ip.h"

#include "../../include/uapi/linux/in.h"

#include "../../include/uapi/linux/tcp.h"

#include "../../include/uapi/linux/udp.h"

//#include "bpf_helpers.h"

#ifndef __section

# define __section(NAME)                  \

           __attribute__((section(NAME), used))

#endif__section("prog")

         //https://www.vultr.com/resources/ipv4-converter/?ip_address=50.50.50.1

         //842150401

int xdp_drop(struct xdp_md *ctx)

{

        void *data_end = (void *)(long)ctx->data_end;

        void *data     = (void *)(long)ctx->data;

        struct ethhdr *eth = data;        if (eth + 1 > data_end) {

                return XDP_PASS;

        }        struct iphdr *iph = data + sizeof(struct ethhdr);        if (iph + 1 > data_end) {

                return XDP_PASS;

        }

        unsigned int ip_src = iph->saddr;

        //printf("%ld\n",htonl(842150401));  network byte order conversion for

        //50.50.50.1

        if(ip_src == 20066866)

        {

                return XDP_DROP;

        }        return XDP_PASS;

}



Good Links

https://docs.cilium.io/en/latest/bpf/

https://medium.com/@fntlnz/load-xdp-programs-using-the-ip-iproute2-command-502043898263
https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf


No comments:

Post a Comment

Featured Post

XDP - Getting Started with XDP (Linux)