Configuring of server iSER - iSCSI Extensions for RDMA on linux Centos 6 and vSphere 6 as iSER client

Hardware Centos 6 linux server: 1xXeon 5504, 12GB MEM, RAID6 ADAPTEC 14 HDD + HOT SPARE 1TB SAS DISKS, Infiniband Mallanox ConnectX®-2 MHQH29C-XTR QR.
Hardware vSphere6 host: 2xXeon x5690, 96 GB MEM, Infiniband Mallanox ConnectX®-2 MHQH29C-XTR QR.
Switches: Infiniband Mallanox IS5023.

Installing Infiniband Support:

yum groupinstall -y "Infiniband Support"
yum install -y opensm

Enabling IPoIB and switch infiniband to connected mode. Enabling iSER support. This Link about differences between connected and datagram modes.

cat /etc/rdma/rdma.conf
# Load IPoIB
IPOIB_LOAD=yes
# Setup connected mode
SET_IPOIB_CM=yes
ISER_LOAD=yes

Enabling and starting Infiniband services.

chkconfig rdma on
chkconfig opensm on
service rdma start
service opensm start

Configuring of two interfaces:

cat /etc/sysconfig/network-scripts/ifcfg-ib0
DEVICE=ib0
HWADDR=80:00:00:48:FE:80:00:00:00:00:00:00:F4:52:14:03:00:CF:45:B3
TYPE=InfiniBand
UUID=ddc8ec72-50fe-42d1-9e2e-1b1758ab6500
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
CONNECTED_MODE=yes
MTU=1500
IPADDR=172.19.254.10
NETMASK=255.255.255.0

cat /etc/sysconfig/network-scripts/ifcfg-ib1
DEVICE=ib1
HWADDR=80:00:00:49:FE:80:00:00:00:00:00:00:F4:52:14:03:00:CF:45:B4
TYPE=InfiniBand
UUID=21a1b0c9-5adc-46dd-b61e-86a7028c204a
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
CONNECTED_MODE=yes
MTU=1500
IPADDR=172.19.254.11
NETMASK=255.255.255.0

Installing iscsi target:

yum install -y scsi-target-utils

Configuring iscsi target:

cat /etc/tgt/targets.conf
default-driver iser
<target iqn.2016-07.ru.kosmosnimki-2>
      backing-store /dev/sda
</target>

Enabling and starting iscsi target:

chkconfig tgtd on
service tgtd start

At first you have to download MLNX-OFED ESX Driver 1.8.3. It is required for Mellanox ConnectX®-2 series.

Removing all native 1.9 verion drivers and reboot.

esxcli software vib remove -f  -n net-ib-cm 
esxcli software vib remove -f  -n net-ib-core
esxcli software vib remove -f -n net-ib-mad 
esxcli software vib remove -f -n net-ib-sa 
esxcli software vib remove -f -n net-ib-umad
esxcli software vib remove -f -n net-mlx4-core
esxcli software vib remove -f -n net-mlx4-en
esxcli software vib remove -f -n net-mlx4-ib
esxcli software vib remove -f -n net-mst
esxcli software vib remove -f -n net-rdma-cm
esxcli software vib remove -f -n scsi-ib-iser
esxcli software vib remove -f -n ib-opensm
esxcli software vib remove -f -n nmlx4-core
esxcli software vib remove -f -n nmlx4-en
esxcli software vib remove -f -n nmlx4-rdma
esxcli software vib remove -f -n scsi-ib-srp
esxcli software vib remove -f -n net-ib-ipoib
esxcli software vib remove -f -n mft
esxcli software vib remove -f -n nmst
esxcli software vib remove -f -n net-ib-addr
reboot

Installing new and reboot:

esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.3.0-10EM-500.0.0.472560.zip --no-sig-check
reboot

Checking:

esxcli software vib list
net-ib-addr                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-ib-cm                      1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-ib-core                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-ib-ipoib                   1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-ib-mad                     1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-ib-sa                      1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-ib-umad                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-mlx4-core                  1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-mlx4-ib                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
net-rdma-cm                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
scsi-ib-iser                   1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
scsi-ib-srp                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20

Also you will see following Storage Adapters

Configuring of two standard switches as shown below:

.

Now you can Add Send Target Server (Centos 6 server) for each IB port. Rescan adapters and format Datastore.

Sequential read: up to ~1.1 GB/s.
Sequential write: up to ~839 MB/s.

*Above values were limited by HDD speed.

Forum Link
Accelerating iSCSI SAN Storage with Mellanox

Profile of the author