How Packets Comes to NIC and Host Machine.

How Packets Comes to NIC and Host Machine.

1. Ring Buffer Contains Start and End Address of Buffer in RAM. TX Ring will contain addresses of Buffer in RAM that contains data to be transmitted. RX Ring will contains address of Buffer in RAM where NIC will place data.
   These rings are present in RAM.
2. TX buffer and RX buffer are are in RAM pointed by TX/RX rings.
3. Now Network Card Register has Location of Rings Buffer in RAM .

Now 1 and 2 can be DMA able buffer , they are called DMA TX/RX ring and DMA TX/RX buffer. Now since RX/TX ring must remain throughout they are made as consistent/coherent DMA type of meory. While Buffers are made streaming/Single DMA type of memory

Reference:
https://homerl.github.io/2018/07/12/nic_info/
https://pdfs.semanticscholar.org/18b1/42b7e8d3ea1549b4f53701d97654179b2616.pdf
(

Development and evaluation of a low-cost scalable architecture for network traffic capture and storage for 10Gbps networks)

Memory Leak using GCC

Detecting Memory Leak using GCC

( without Valgrind)

#include

int main()

{

  int *prt =(int*) malloc(sizeof(int));;

  return 0;

}

08:46:32alok@~$ gcc memory_leak.c 

08:46:51alok@~$ ./a.out 

08:46:52alok@~$ gcc -fsanitize=address -g memory_leak.c 

08:46:59alok@~$ ./a.out 

====================================================

==4221==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 4 byte(s) in 1 object(s) allocated from:

    #0 0x7fe31f218acb in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/liblsan.so.0+0xeacb)

    #1 0x558bd3ad271b in main /home/alok/memory_leak.c:4

    #2 0x7fe31ee3ab96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: LeakSanitizer: 4 byte(s) leaked in 1 allocation(s).

RDMA Verb Programming on SoftRoCE

Installing SoftRoCE on Two VMs on virtualbox for

RDMA Verb Programming

root@embsys-VirtualBox:~/softroce# git clone https://github.com/SoftRoCE/rxe-dev                                         

[Around 1.1 GB Downoad]

Installation

1. Kernel Older than 4.8

https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home

2. Kernel 4.9 has inbuilt SoftROCE

https://community.mellanox.com/docs/DOC-2184

Sources

Kernel Space Driver

• Github: https://github.com/SoftRoCE/rxe-dev.git (this repository)
• Active Branch: rxe_submission_v18

User Space Library

• Github: https://github.com/SoftRoCE/librxe-dev.git
• Current Version: librxe-1.0.0

Installing Kernel Space Driver

root@embsys-VirtualBox:~/softroce/rxe-dev# git checkout rxe_submission_v18

Checking out files: 100% (32626/32626), done.

Branch rxe_submission_v18 set up to track remote branch rxe_submission_v18 from origin.

Switched to a new branch 'rxe_submission_v18'

Prerequsitive Packages

apt-get install libncurses5-dev libncursesw5-dev  libssl-dev build-essential make bc

Copy current Running Config

root@embsys-VirtualBox:~/softroce/rxe-dev# uname -r

4.10.0-42-generic

root@embsys-VirtualBox:~/softroce/rxe-dev# cp /boot/config-4.10.0-42-generic .config

Select ROCE package from Source and Compile Kernel

root@embsys-VirtualBox:~/softroce/rxe-dev# make menuconfig

Enter menuconfig and press "/" and enter "RXE" to search for the relevant options

Then press 1 to select the first search result and select it

After selecting the save and save directly into. Config has been elected exit

Select - >Save and Exit

Start compilation

Make –j 4

Compilation Progress …Took 1:15 Hrs

sudo make modules_install

Update New Kernel in grub

root@embsys-VirtualBox:/roce/softroce/rxe-dev# make install

sh ./arch/x86/boot/install.sh 4.7.0-rc3+ arch/x86/boot/bzImage \

       System.map "/boot"

run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

update-initramfs: Generating /boot/initrd.img-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/pm-utils 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/update-notifier 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

Generating grub configuration file ...

Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.

Found linux image: /boot/vmlinuz-4.10.0-42-generic

Found initrd image: /boot/initrd.img-4.10.0-42-generic

Found linux image: /boot/vmlinuz-4.10.0-38-generic

Found initrd image: /boot/initrd.img-4.10.0-38-generic

Found linux image: /boot/vmlinuz-4.7.0-rc3+

Found initrd image: /boot/initrd.img-4.7.0-rc3+

Found memtest86+ image: /boot/memtest86+.elf

Found memtest86+ image: /boot/memtest86+.bin

done

Reboot, and select OS to Following which has 4.7 Kernel.

sudo make headers_install INSTALL_HDR_PATH=/usr

( No space between = sign)

Verify using  ( config feature support should be enabled in kernel)

embsys@embsys-VirtualBox:~$ zgrep -a "RDMA" /proc/config.gz

CONFIG_RDS_RDMA=m

CONFIG_CARDMAN_4000=m

CONFIG_CARDMAN_4040=m

# CONFIG_INFINIBAND_OCRDMA is not set

CONFIG_RDMA_RXE=m

CONFIG_INFINIBAND_RDMAVT=m

CONFIG_SUNRPC_XPRT_RDMA=m

Installing User Space driver/App

apt-get install libibverbs-dev libswitch-perl rdmacm-utils ibverbs-utils

git clone https://github.com/SoftRoCE/librxe-dev.git

cd librxe-dev

Compile the librxe

./configure --libdir=/usr/lib64 -prefix=

Make

Make install

Make symbolic Links

sudo ln -s /usr/lib64/librxe.a /usr/lib/librxe.a

sudo ln -s /usr/lib64/librxe.la /usr/lib/librxe.la

sudo ln -s /usr/lib64/librxe-rdmav2.so /usr/lib/librxe-rdmav2.so

sudo ln -s /usr/lib64/librxe.so /usr/lib/librxe.so

Once install check the status before loading rxe module

rxe_cfg status

Binding driver to Ethernet

Rxe_cfg add

Modules Loaded

root@embsys-VirtualBox:/roce/softroce/librxe-dev# lsmod | grep rxe

rdma_rxe              102400  0

ip6_udp_tunnel         16384  1 rdma_rxe

udp_tunnel             16384  1 rdma_rxe

ib_core               196608  6 rdma_cm,ib_cm,iw_cm,ib_uverbs,rdma_rxe,rdma_ucm

Dmesg

Check RDMA Device using ibv_devices and ibv_devinfo

root@embsys-VirtualBox:/roce/softroce/librxe-dev# ibv_devices    device                 node GUID    ------              ----------------    rxe0                0a0027fffe28b552 root@embsys-VirtualBox:/roce/softroce/librxe-dev# ibv_devinfo hca_id: rxe0        transport:                      InfiniBand (0)        fw_ver:                         0.0.0        node_guid:                      0a00:27ff:fe28:b552        sys_image_guid:                 0000:0000:0000:0000        vendor_id:                      0x0000        vendor_part_id:                 0        hw_ver:                         0x0        phys_port_cnt:                  1          port: 1                state:                  PORT_ACTIVE (4)                max_mtu:                4096 (5)                active_mtu:             1024 (3)                sm_lid:                 0                port_lid:               0                port_lmc:               0x00                link_layer:             Ethernet

( A interface with Bridge Connection between Two is required)

https://www.youtube.com/watch?v=Vc1gSSyzDJU

* Update
One of the Nice project , for understanding RDMA Programmiing using softroce is
https://github.com/haggaie/rdma-experiment
you my fork for two VM variant
https://github.com/alokprasad/rdma-experiment