RDMA Verb Programming on SoftRoCE

Installing SoftRoCE on Two VMs on virtualbox for

RDMA Verb Programming

root@embsys-VirtualBox:~/softroce# git clone https://github.com/SoftRoCE/rxe-dev                                         

[Around 1.1 GB Downoad]

Installation

1. Kernel Older than 4.8

https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home

2. Kernel 4.9 has inbuilt SoftROCE

https://community.mellanox.com/docs/DOC-2184

Sources

Kernel Space Driver

• Github: https://github.com/SoftRoCE/rxe-dev.git (this repository)
• Active Branch: rxe_submission_v18

User Space Library

• Github: https://github.com/SoftRoCE/librxe-dev.git
• Current Version: librxe-1.0.0

Installing Kernel Space Driver

root@embsys-VirtualBox:~/softroce/rxe-dev# git checkout rxe_submission_v18

Checking out files: 100% (32626/32626), done.

Branch rxe_submission_v18 set up to track remote branch rxe_submission_v18 from origin.

Switched to a new branch 'rxe_submission_v18'

Prerequsitive Packages

apt-get install libncurses5-dev libncursesw5-dev  libssl-dev build-essential make bc

Copy current Running Config

root@embsys-VirtualBox:~/softroce/rxe-dev# uname -r

4.10.0-42-generic

root@embsys-VirtualBox:~/softroce/rxe-dev# cp /boot/config-4.10.0-42-generic .config

Select ROCE package from Source and Compile Kernel

root@embsys-VirtualBox:~/softroce/rxe-dev# make menuconfig

Enter menuconfig and press "/" and enter "RXE" to search for the relevant options

Then press 1 to select the first search result and select it

After selecting the save and save directly into. Config has been elected exit

Select - >Save and Exit

Start compilation

Make –j 4

Compilation Progress …Took 1:15 Hrs

sudo make modules_install

Update New Kernel in grub

root@embsys-VirtualBox:/roce/softroce/rxe-dev# make install

sh ./arch/x86/boot/install.sh 4.7.0-rc3+ arch/x86/boot/bzImage \

       System.map "/boot"

run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

update-initramfs: Generating /boot/initrd.img-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/pm-utils 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/update-notifier 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.7.0-rc3+ /boot/vmlinuz-4.7.0-rc3+

Generating grub configuration file ...

Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.

Found linux image: /boot/vmlinuz-4.10.0-42-generic

Found initrd image: /boot/initrd.img-4.10.0-42-generic

Found linux image: /boot/vmlinuz-4.10.0-38-generic

Found initrd image: /boot/initrd.img-4.10.0-38-generic

Found linux image: /boot/vmlinuz-4.7.0-rc3+

Found initrd image: /boot/initrd.img-4.7.0-rc3+

Found memtest86+ image: /boot/memtest86+.elf

Found memtest86+ image: /boot/memtest86+.bin

done

Reboot, and select OS to Following which has 4.7 Kernel.

sudo make headers_install INSTALL_HDR_PATH=/usr

( No space between = sign)

Verify using  ( config feature support should be enabled in kernel)

embsys@embsys-VirtualBox:~$ zgrep -a "RDMA" /proc/config.gz

CONFIG_RDS_RDMA=m

CONFIG_CARDMAN_4000=m

CONFIG_CARDMAN_4040=m

# CONFIG_INFINIBAND_OCRDMA is not set

CONFIG_RDMA_RXE=m

CONFIG_INFINIBAND_RDMAVT=m

CONFIG_SUNRPC_XPRT_RDMA=m

Installing User Space driver/App

apt-get install libibverbs-dev libswitch-perl rdmacm-utils ibverbs-utils

git clone https://github.com/SoftRoCE/librxe-dev.git

cd librxe-dev

Compile the librxe

./configure --libdir=/usr/lib64 -prefix=

Make

Make install

Make symbolic Links

sudo ln -s /usr/lib64/librxe.a /usr/lib/librxe.a

sudo ln -s /usr/lib64/librxe.la /usr/lib/librxe.la

sudo ln -s /usr/lib64/librxe-rdmav2.so /usr/lib/librxe-rdmav2.so

sudo ln -s /usr/lib64/librxe.so /usr/lib/librxe.so

Once install check the status before loading rxe module

rxe_cfg status

Binding driver to Ethernet

Rxe_cfg add

Modules Loaded

root@embsys-VirtualBox:/roce/softroce/librxe-dev# lsmod | grep rxe

rdma_rxe              102400  0

ip6_udp_tunnel         16384  1 rdma_rxe

udp_tunnel             16384  1 rdma_rxe

ib_core               196608  6 rdma_cm,ib_cm,iw_cm,ib_uverbs,rdma_rxe,rdma_ucm

Dmesg

Check RDMA Device using ibv_devices and ibv_devinfo

root@embsys-VirtualBox:/roce/softroce/librxe-dev# ibv_devices    device                 node GUID    ------              ----------------    rxe0                0a0027fffe28b552 root@embsys-VirtualBox:/roce/softroce/librxe-dev# ibv_devinfo hca_id: rxe0        transport:                      InfiniBand (0)        fw_ver:                         0.0.0        node_guid:                      0a00:27ff:fe28:b552        sys_image_guid:                 0000:0000:0000:0000        vendor_id:                      0x0000        vendor_part_id:                 0        hw_ver:                         0x0        phys_port_cnt:                  1          port: 1                state:                  PORT_ACTIVE (4)                max_mtu:                4096 (5)                active_mtu:             1024 (3)                sm_lid:                 0                port_lid:               0                port_lmc:               0x00                link_layer:             Ethernet

( A interface with Bridge Connection between Two is required)

https://www.youtube.com/watch?v=Vc1gSSyzDJU

* Update
One of the Nice project , for understanding RDMA Programmiing using softroce is
https://github.com/haggaie/rdma-experiment
you my fork for two VM variant
https://github.com/alokprasad/rdma-experiment