======= Deploying ScaleIO 1.3======= //This article is exploring the process of creation block storage based on %%ScaleIO%% 1.3. %%ScaleIO%% was deployed on three Linux %%CentOS%% 7 servers with 14 local disks and several %%VmWare%% 5.5 hosts as a clients. All hosts connected with Ethernet 10Gbit/s. // ==== Introduction. ==== %%EMC ScaleIO%% is a software-defined storage product from %%EMC%% Corporation that creates a server-based storage area network (SAN) from local application server storage, converting direct-attached storage into shared block storage. It uses existing host-based internal storage to create a scalable, high-performance, low-cost server SAN. %%EMC promotes its ScaleIO%% server storage-area network software as a way to converge computing resources and commodity storage into a “single-layer architecture.”\\ %%ScaleIO%% can scale from three compute/storage nodes to over 1,000 nodes that can drive up to 240 million IOPS of performance. Developers can deploy the %%ScaleIO%% software on-prem commodity infrastructure or in the cloud and then port their applications back into a production %%ScaleIO%% instance.\\ Al last %%EMC DAY ROADSHOW 2016%% in Moscow was announced that %%ScaleIO%% will be %%OpenSource%%*. ==== Scheme and Equipments. ==== **Scheme** {{:ru:jobs:scaleios_forart.jpg?800|500}} LACP bonding is used for connecting %%ScaleIO%% nodes to Ethernet switches. Ethernet switches configured with vPC.\\ **Equipments** %%ScaleIO%% nodes: Proc - Intel Xeon E5520/5620\\ Memory - 16 GB\\ OS DISK - 2X40GB (RAID1)\\ DISK - 14x4TB\\ ==== Configuration ==== **Shortly about initial configuration of %%CentOS%% on test-scaleio-nodes** **1. Configure UDEV rules for each of disks. For example create links in /dev as 1-hdd,2-hdd,..,14-hdd. Without links in a case of faults of one or more hdd disks and subsequent reboot sds devices will be associated with inappropriate HDDs.**\\ 2. All ethernet ports in bond (mode 4). VLAN2 - management, VLAN5 - data\\ 3. Passwordless access between nodes for user root. The key shoud be created on one node and then copy to other to /root/.ssh/. ssh-keygen -t dsa (creation of passwordless key) cd /root/.ssh cat id_dsa.pub >> authorized_keys chown root.root authorized_keys chmod 600 authorized_keys echo "StrictHostKeyChecking no" > config 4. Don't forget to write all names hosts in /etc/hosts.\\ 5. Configure ntpd on all nodes\\ 6. %%IPTABLES/Firewall%%\\ Allow ports sshd and %%ScaleIO%% tcp incoming ports 6611,9011,7072. (use official documentation [[https://www.emc.com/collateral/technicaldocument/emc-scaleio-installation-guide.pdf|EMC]]) 7. Disable selinux\\ 8. Configuration of sysctl.conf net.ipv4.tcp_timestamps=0 net.ipv4.tcp_sack=0 net.core.netdev_max_backlog=250000 net.core.rmem_max=8388608 net.core.wmem_max=8388608 net.core.rmem_default=8388608 net.core.wmem_default=8388608 net.core.optmem_max=8388608 net.ipv4.tcp_mem=8388608 8388608 8388608 net.ipv4.tcp_rmem=4096 87380 8388608 net.ipv4.tcp_wmem=4096 65536 8388608 9. Add max open files to /etc/security/limits.conf (depends on your requirements) * hard nofile 1000000 * soft nofile 1000000 ==== ScaleIO installation === 1. Downloading [[http://www.emc.com/products-solutions/trial-software-download/scaleio.htm|Links for download]]. Some words about Linux packages. All packages comes as rpm and in case of using %%Spacewalk%% you can create an updating the %%ScaleIO%% repo.\\ **%%Installing on ScaleIO nodes%%** Nodes test-scaleio-node1-2 will be MDM and test-scaleio-node3 will be Tie-Breaker. Installing required packages:\\ yum install libaio numactl mutt bash-completion python -y Installing package for MDM nodes: rpm -ivh /downloads/EMC-ScaleIO-mdm-1.32-403.2.el7.x86_64.rpm Installing package for Tie-Breaker node: rpm -ivh /downloads/EMC-ScaleIO-tb-1.32-403.2.el7.x86_64.rpm Installing packages on all nodes rpm -ivh /downloads/EMC-ScaleIO-callhome-1.32-403.2.el7.x86_64.rpm rpm -ivh /downloads/EMC-ScaleIO-sds-1.32-403.2.el7.x86_64.rpm" **%%Installing on VmWare hosts%%** esxcli software acceptance set --level=PartnerSupported esxcli software vib install -d /tmp/sdc-1.32.343.0-esx5.5.zip reboot ==== Cluster configuration==== **1. Starting services** For MDM nodes: systemctl start mdm.service systemctl enable mdm.service For Tie-Breaker node: systemctl start tb.service systemctl enable tb.service For all nodes: systemctl start sds.service callhome.service systemctl enable sds.service callhome.service **2. Configuring MDM cluster** ** All action performing on test-scaleio-node1 ** Login to first mdm to change password and accept licence: scli --add_primary_mdm --primary_mdm_ip 10.255.254.100 --accept_license scli --login --username admin --password admin scli --set_password --mdm_ip 10.255.254.100 --old_password admin --new_password YourPass scli --login --username admin --password YourPass #Required to perform any operations. Add secondary MDM and Tie-Breaker to cluster: scli --add_secondary_mdm --secondary_mdm_ip 10.255.254.101 scli --add_tb --tb_ip 10.255.254.102 Checking cluster: scli --query_cluster Switch to cluster mode: scli --switch_to_cluster_mode scli --query_cluster Now you can also use GUI for sds configuration. Install on your Windows PC %% EMC ScaleIO GUI%%.\\ **3. Configuring SDS.** Setup protection domain and storage pool scli --add_protection_domain --protection_domain_name test scli --add_storage_pool --protection_domain_name test --storage_pool_name test01 Configuring SDSs and devices: scli --add_sds --sds_ip 10.255.254.100 --protection_domain_name test --storage_pool_name test01 --device_path /dev/1-hdd,/dev/2-hdd,/dev/3-hdd,/dev/4-hdd,/dev/5-hdd,/dev/6-hdd,/dev/7-hdd,/dev/8-hdd,/dev/9-hdd,/dev/10-hdd,/dev/11-hdd,/dev/12-hdd,/dev/13-hdd,/dev/14-hdd --device_name sds01100,sds02100,sds03100,sds04100,sds05100,sds06100,sds07100,sds08100,sds09100,sds10100,sds11100,sds12100,sds13100,sds14100 scli --add_sds --sds_ip 10.255.254.101 --protection_domain_name test --storage_pool_name test01 --device_path /dev/1-hdd,/dev/2-hdd,/dev/3-hdd,/dev/4-hdd,/dev/5-hdd,/dev/6-hdd,/dev/7-hdd,/dev/8-hdd,/dev/9-hdd,/dev/10-hdd,/dev/11-hdd,/dev/12-hdd,/dev/13-hdd,/dev/14-hdd --device_name ds01101,sds02101,sds03101,sds04101,sds05101,sds06101,sds07101,sds08101,sds09101,sds10101,sds11101,sds12101,sds13101,sds14101 scli --add_sds --sds_ip 10.255.254.102 --protection_domain_name test --storage_pool_name test01 --device_path /dev/1-hdd,/dev/2-hdd,/dev/3-hdd,/dev/4-hdd,/dev/5-hdd,/dev/6-hdd,/dev/7-hdd,/dev/8-hdd,/dev/9-hdd,/dev/10-hdd,/dev/11-hdd,/dev/12-hdd,/dev/13-hdd,/dev/14-hdd --device_name ds01102,sds02102,sds03102,sds04102,sds05102,sds06102,sds07102,sds08102,sds09102,sds10201,sds11102,sds12102,sds13102,sds14102 Setup rebalance and spare policy: scli --set_rebalance_mode --protection_domain_name test --storage_pool_name test01 --enable_rebalance scli --set_rebuild_policy --protection_domain_name test --storage_pool_name test01 --policy no_limit scli --modify_spare_policy --protection_domain_name test --storage_pool_name test01 --spare_percentage 34 --i_am_sure Spare policy was installed with 34% spare capacity space (for 3 SDS nodes).\\ Without that you will face with warning - "Configured spare capacity is smaller than largest fault unit" Also you can setup read cache size: scli --set_sds_rmcache_size --protection_domain_name test --rmcache_size_mb 2048 **4. Creating volume:** scli --add_volume --protection_domain_name test --storage_pool_name test01 --size_gb YOUR_SIZE_IN_MB --volume_name test_vol0 **5. Mapping volume to clients (%%VmWare%% vSphere hosts):** scli --map_volume_to_sdc --volume_name test_vol0 --sdc_ip 10.255.254.200 Where 10.255.254.200 is IP of first vSphere ESX host. ==== Configuring VmWare ESX hosts==== Generate some UUID anywhere. This UUID will be ESX hosts client identification number.\\ From ESX nodes: esxcli system module parameters set -m scini -p "IoctlIniGuidStr=12A97DE0-7CED-440C-AF9A-1BF51CAA699F IoctlMdmIPStr=10.255.254.100,10.255.254.101" esxcli system module load -m scini reboot (it is not always required) Now you can perform"Rescan ALL" in configuration -> Storage Adapter. And you will see some Fibre Channel device and you will can add storage. ==== Testing ==== There are no any test results yet. ==== Conclusion. ==== Despite absence of results of tests %%ScaleIO%% is very productive and reliable. \\ But performance suffered if one of nodes has been rebooted . (You can setup re-balance limit to reduce influence to performance) Also we faced with situation when we had to wait for full synchronization to start using volume.\\ Enjoy! ==== About author ==== [[https://www.linkedin.com/pub/alexey-vyrodov/59/976/16b|Profile]] of the author ==== Links. ==== 1. [[https://en.wikipedia.org/wiki/EMC_ScaleIO|WiKi]]\\ 2. [[https://www.emc.com/collateral/technicaldocument/emc-scaleio-installation-guide.pdf|Official user guide]]\\