✎ en:jobs:iser_vsphere6 [IT Notes about: Juniper, Cisco, Checkpoint, FreeBSD, Linux, Windows, VmWare, VDI, Cloud....]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======= Configuring of server iSER - iSCSI Extensions for RDMA on linux Centos 6 and vSphere 6 as iSER client=======

==== Hardware ====

Hardware Centos 6 linux server: 1xXeon 5504, 12GB MEM, RAID6 ADAPTEC 14 HDD + HOT SPARE 1TB SAS DISKS, Infiniband Mallanox %%ConnectX®-2%% MHQH29C-XTR QR.\\
Hardware vSphere6 host: 2xXeon x5690, 96 GB MEM, Infiniband Mallanox %%ConnectX®-2%% MHQH29C-XTR QR.\\
Switches: Infiniband Mallanox IS5023.\\

 

==== Configuring iSER server on Centos 6 ====

Installing Infiniband Support:\\

  
  yum groupinstall -y "Infiniband Support"
  yum install -y opensm

Enabling %%IPoIB%% and switch infiniband to connected mode. Enabling iSER support. This [[https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt|Link]] about differences between connected and datagram modes. 

  cat /etc/rdma/rdma.conf
  # Load IPoIB
  IPOIB_LOAD=yes
  # Setup connected mode
  SET_IPOIB_CM=yes
  ISER_LOAD=yes

Enabling and starting Infiniband services.

  chkconfig rdma on
  chkconfig opensm on
  service rdma start
  service opensm start

Configuring of two interfaces:\\
*vSphere ESXi maximum mtu size is 9000 bytes. Required to configure the same mtu on Linux Centos server. 

  cat /etc/sysconfig/network-scripts/ifcfg-ib0
  DEVICE=ib0
  HWADDR=80:00:00:48:FE:80:00:00:00:00:00:00:F4:52:14:03:00:CF:45:B3
  TYPE=InfiniBand
  UUID=ddc8ec72-50fe-42d1-9e2e-1b1758ab6500
  ONBOOT=yes
  NM_CONTROLLED=no
  BOOTPROTO=none
  CONNECTED_MODE=yes
  MTU=1500
  IPADDR=172.19.254.10
  NETMASK=255.255.255.0

  cat /etc/sysconfig/network-scripts/ifcfg-ib1
  DEVICE=ib1
  HWADDR=80:00:00:49:FE:80:00:00:00:00:00:00:F4:52:14:03:00:CF:45:B4
  TYPE=InfiniBand
  UUID=21a1b0c9-5adc-46dd-b61e-86a7028c204a
  ONBOOT=yes
  NM_CONTROLLED=no
  BOOTPROTO=none
  CONNECTED_MODE=yes
  MTU=1500
  IPADDR=172.19.254.11
  NETMASK=255.255.255.0

Installing iscsi target:\\

  yum install -y scsi-target-utils

Configuring iscsi target:\\

  cat /etc/tgt/targets.conf
  default-driver iser
  <target iqn.2016-07.ru.kosmosnimki-2>
        backing-store /dev/sda
  </target>

Enabling and starting iscsi target:\\
  
  chkconfig tgtd on
  service tgtd start

==== Configuring iSER client on vSphere 6 ====
At first you have to download MLNX-OFED ESX Driver 1.8.3. It is required for Mellanox %%ConnectX®-2%% series.\\

Removing all native 1.9 verion drivers and reboot.\\
  
  esxcli software vib remove -f  -n net-ib-cm 
  esxcli software vib remove -f  -n net-ib-core
  esxcli software vib remove -f -n net-ib-mad 
  esxcli software vib remove -f -n net-ib-sa 
  esxcli software vib remove -f -n net-ib-umad
  esxcli software vib remove -f -n net-mlx4-core
  esxcli software vib remove -f -n net-mlx4-en
  esxcli software vib remove -f -n net-mlx4-ib
  esxcli software vib remove -f -n net-mst
  esxcli software vib remove -f -n net-rdma-cm
  esxcli software vib remove -f -n scsi-ib-iser
  esxcli software vib remove -f -n ib-opensm
  esxcli software vib remove -f -n nmlx4-core
  esxcli software vib remove -f -n nmlx4-en
  esxcli software vib remove -f -n nmlx4-rdma
  esxcli software vib remove -f -n scsi-ib-srp
  esxcli software vib remove -f -n net-ib-ipoib
  esxcli software vib remove -f -n mft
  esxcli software vib remove -f -n nmst
  esxcli software vib remove -f -n net-ib-addr
  reboot

Installing new and reboot:\\
  esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.3.0-10EM-500.0.0.472560.zip --no-sig-check
  reboot

Checking:\\
  esxcli software vib list
  net-ib-addr                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-ib-cm                      1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-ib-core                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-ib-ipoib                   1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-ib-mad                     1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-ib-sa                      1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-ib-umad                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-mlx4-core                  1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-mlx4-ib                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  net-rdma-cm                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  scsi-ib-iser                   1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20
  scsi-ib-srp                    1.8.3.0-1OEM.500.0.0.472560           Mellanox  PartnerSupported  2016-07-20

Also you will see following Storage Adapters\\

 {{:ru:jobs:vsphere_iser.jpg?800|600}}\\

Configuring of two standard switches as shown below:\\

{{:ru:jobs:vsphere_net.jpg?800|600}}.\\

Now you can Add Send Target Server (Centos 6 server) for each IB port. Rescan adapters and format Datastore.

==== Performance  ====

Sequential read: up to ~1.1 GB/s.\\
Sequential write: up to ~839 MB/s.\\

*Above values were limited by HDD speed.  


==== Links  ====
[[https://forums.servethehome.com/index.php?threads/connectx-2-and-esxi-6-0.5382/|Forum Link]]\\
[[https://www.mellanox.com/related-docs/case_studies/CS_StarWind2.pdf| Accelerating iSCSI SAN Storage with Mellanox]]

==== About author  ====
[[https://www.linkedin.com/pub/alexey-vyrodov/59/976/16b|Profile]] of the author