======= Configuring of server iSER - iSCSI Extensions for RDMA on linux Centos 6 and vSphere 6 as iSER client======= ==== Hardware ==== Hardware Centos 6 linux server: 1xXeon 5504, 12GB MEM, RAID6 ADAPTEC 14 HDD + HOT SPARE 1TB SAS DISKS, Infiniband Mallanox %%ConnectX®-2%% MHQH29C-XTR QR.\\ Hardware vSphere6 host: 2xXeon x5690, 96 GB MEM, Infiniband Mallanox %%ConnectX®-2%% MHQH29C-XTR QR.\\ Switches: Infiniband Mallanox IS5023.\\ ==== Configuring iSER server on Centos 6 ==== Installing Infiniband Support:\\ yum groupinstall -y "Infiniband Support" yum install -y opensm Enabling %%IPoIB%% and switch infiniband to connected mode. Enabling iSER support. This [[https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt|Link]] about differences between connected and datagram modes. cat /etc/rdma/rdma.conf # Load IPoIB IPOIB_LOAD=yes # Setup connected mode SET_IPOIB_CM=yes ISER_LOAD=yes Enabling and starting Infiniband services. chkconfig rdma on chkconfig opensm on service rdma start service opensm start Configuring of two interfaces:\\ cat /etc/sysconfig/network-scripts/ifcfg-ib0 DEVICE=ib0 HWADDR=80:00:00:48:FE:80:00:00:00:00:00:00:F4:52:14:03:00:CF:45:B3 TYPE=InfiniBand UUID=ddc8ec72-50fe-42d1-9e2e-1b1758ab6500 ONBOOT=yes NM_CONTROLLED=no BOOTPROTO=none CONNECTED_MODE=yes MTU=1500 IPADDR=172.19.254.10 NETMASK=255.255.255.0 cat /etc/sysconfig/network-scripts/ifcfg-ib1 DEVICE=ib1 HWADDR=80:00:00:49:FE:80:00:00:00:00:00:00:F4:52:14:03:00:CF:45:B4 TYPE=InfiniBand UUID=21a1b0c9-5adc-46dd-b61e-86a7028c204a ONBOOT=yes NM_CONTROLLED=no BOOTPROTO=none CONNECTED_MODE=yes MTU=1500 IPADDR=172.19.254.11 NETMASK=255.255.255.0 Installing iscsi target:\\ yum install -y scsi-target-utils Configuring iscsi target:\\ cat /etc/tgt/targets.conf default-driver iser backing-store /dev/sda Enabling and starting iscsi target:\\ chkconfig tgtd on service tgtd start ==== Configuring iSER client on vSphere 6 ==== At first you have to download MLNX-OFED ESX Driver 1.8.3. It is required for Mellanox %%ConnectX®-2%% series.\\ Removing all native 1.9 verion drivers and reboot.\\ esxcli software vib remove -f -n net-ib-cm esxcli software vib remove -f -n net-ib-core esxcli software vib remove -f -n net-ib-mad esxcli software vib remove -f -n net-ib-sa esxcli software vib remove -f -n net-ib-umad esxcli software vib remove -f -n net-mlx4-core esxcli software vib remove -f -n net-mlx4-en esxcli software vib remove -f -n net-mlx4-ib esxcli software vib remove -f -n net-mst esxcli software vib remove -f -n net-rdma-cm esxcli software vib remove -f -n scsi-ib-iser esxcli software vib remove -f -n ib-opensm esxcli software vib remove -f -n nmlx4-core esxcli software vib remove -f -n nmlx4-en esxcli software vib remove -f -n nmlx4-rdma esxcli software vib remove -f -n scsi-ib-srp esxcli software vib remove -f -n net-ib-ipoib esxcli software vib remove -f -n mft esxcli software vib remove -f -n nmst esxcli software vib remove -f -n net-ib-addr reboot Installing new and reboot:\\ esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.3.0-10EM-500.0.0.472560.zip --no-sig-check reboot Checking:\\ esxcli software vib list net-ib-addr 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-ib-cm 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-ib-core 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-ib-ipoib 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-ib-mad 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-ib-sa 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-ib-umad 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-mlx4-core 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-mlx4-ib 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 net-rdma-cm 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 scsi-ib-iser 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 scsi-ib-srp 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20 Also you will see following Storage Adapters\\ {{:ru:jobs:vsphere_iser.jpg?800|600}}\\ Configuring of two standard switches as shown below:\\ {{:ru:jobs:vsphere_net.jpg?800|600}}.\\ Now you can Add Send Target Server (Centos 6 server) for each IB port. Rescan adapters and format Datastore. ==== Performance ==== Sequential read: up to ~1.1 GB/s.\\ Sequential write: up to ~839 MB/s.\\ *Above values were limited by HDD speed. ==== Links ==== [[https://forums.servethehome.com/index.php?threads/connectx-2-and-esxi-6-0.5382/|Forum Link]]\\ [[https://www.mellanox.com/related-docs/case_studies/CS_StarWind2.pdf| Accelerating iSCSI SAN Storage with Mellanox]] ==== About author ==== [[https://www.linkedin.com/pub/alexey-vyrodov/59/976/16b|Profile]] of the author