Configuring of server iSER - iSCSI Extensions for RDMA on linux Centos 6 and vSphere 6 as iSER client
Hardware
Hardware Centos 6 linux server: 1xXeon 5504, 12GB MEM, RAID6 ADAPTEC 14 HDD + HOT SPARE 1TB SAS DISKS, Infiniband Mallanox ConnectX®-2 MHQH29C-XTR QR.
Hardware vSphere6 host: 2xXeon x5690, 96 GB MEM, Infiniband Mallanox ConnectX®-2 MHQH29C-XTR QR.
Switches: Infiniband Mallanox IS5023.
Configuring iSER server on Centos 6
Installing Infiniband Support:
yum groupinstall -y "Infiniband Support"
yum install -y opensm
Enabling IPoIB and switch infiniband to connected mode. Enabling iSER support. This Link about differences between connected and datagram modes.
cat /etc/rdma/rdma.conf
# Load IPoIB
IPOIB_LOAD=yes
# Setup connected mode
SET_IPOIB_CM=yes
ISER_LOAD=yes
Enabling and starting Infiniband services.
chkconfig rdma on
chkconfig opensm on
service rdma start
service opensm start
Configuring of two interfaces:
cat /etc/sysconfig/network-scripts/ifcfg-ib0
DEVICE=ib0
HWADDR=80:00:00:48:FE:80:00:00:00:00:00:00:F4:52:14:03:00:CF:45:B3
TYPE=InfiniBand
UUID=ddc8ec72-50fe-42d1-9e2e-1b1758ab6500
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
CONNECTED_MODE=yes
MTU=1500
IPADDR=172.19.254.10
NETMASK=255.255.255.0
cat /etc/sysconfig/network-scripts/ifcfg-ib1
DEVICE=ib1
HWADDR=80:00:00:49:FE:80:00:00:00:00:00:00:F4:52:14:03:00:CF:45:B4
TYPE=InfiniBand
UUID=21a1b0c9-5adc-46dd-b61e-86a7028c204a
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
CONNECTED_MODE=yes
MTU=1500
IPADDR=172.19.254.11
NETMASK=255.255.255.0
Installing iscsi target:
yum install -y scsi-target-utils
Configuring iscsi target:
cat /etc/tgt/targets.conf
default-driver iser
<target iqn.2016-07.ru.kosmosnimki-2>
backing-store /dev/sda
</target>
Enabling and starting iscsi target:
chkconfig tgtd on
service tgtd start
Configuring iSER client on vSphere 6
At first you have to download MLNX-OFED ESX Driver 1.8.3. It is required for Mellanox ConnectX®-2 series.
Removing all native 1.9 verion drivers and reboot.
esxcli software vib remove -f -n net-ib-cm
esxcli software vib remove -f -n net-ib-core
esxcli software vib remove -f -n net-ib-mad
esxcli software vib remove -f -n net-ib-sa
esxcli software vib remove -f -n net-ib-umad
esxcli software vib remove -f -n net-mlx4-core
esxcli software vib remove -f -n net-mlx4-en
esxcli software vib remove -f -n net-mlx4-ib
esxcli software vib remove -f -n net-mst
esxcli software vib remove -f -n net-rdma-cm
esxcli software vib remove -f -n scsi-ib-iser
esxcli software vib remove -f -n ib-opensm
esxcli software vib remove -f -n nmlx4-core
esxcli software vib remove -f -n nmlx4-en
esxcli software vib remove -f -n nmlx4-rdma
esxcli software vib remove -f -n scsi-ib-srp
esxcli software vib remove -f -n net-ib-ipoib
esxcli software vib remove -f -n mft
esxcli software vib remove -f -n nmst
esxcli software vib remove -f -n net-ib-addr
reboot
Installing new and reboot:
esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.3.0-10EM-500.0.0.472560.zip --no-sig-check
reboot
Checking:
esxcli software vib list
net-ib-addr 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-ib-cm 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-ib-core 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-ib-ipoib 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-ib-mad 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-ib-sa 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-ib-umad 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-mlx4-core 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-mlx4-ib 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
net-rdma-cm 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
scsi-ib-iser 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
scsi-ib-srp 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2016-07-20
Also you will see following Storage Adapters
Configuring of two standard switches as shown below:
.
Now you can Add Send Target Server (Centos 6 server) for each IB port. Rescan adapters and format Datastore.
Sequential read: up to ~1.1 GB/s.
Sequential write: up to ~839 MB/s.
*Above values were limited by HDD speed.
Links
About author