ZFS deduplication is very atractive thing that decrease space usage for storing VM. But how can we use it with openstack.
The answer is ceph rbd over ZVOL.
This article will explore procees of creation stend for test of HA openstack with rbd over zvol.

Installing task includes:

1. Installing ceph rbd on all nodes.

2. Installing HA openstack components which include:

2.1. Installing of mysql cluster
2.2. Installing of RabbitMQ cluster
2.3. Installing of HA OpenStack Identity*
2.4. Installing of HA Cinder*

* pcs cluster will be required.

This article will not concern details of OS tuning and network configuration because it is too early with alfa-release.
Hardware configuration.
Two hdd for OS (RAID1), three HDD for data, 2xXeon56XX, 48 GB MEM.
Software configuration.
OS Centos 7.1. Selinux and firewall were disabled. Ssh passwordless access was configured on all nodes. Data disks ware assembled with (RAIDz) base on zvol (zfs for linux). Deduplication switch on. Follow vdev was created /dev/zvol/data/vdata
All nodes have the same /etc/hosts file:

 192.168.6.214 node1
 192.168.6.215 node2
 192.168.6.216 node3
 192.168.6.240 shareIP

Repo below should be installed with zfs on linux instalation. Anywhere:

yum localinstall --nogpgcheck http://mirror.yandex.ru/epel/7/x86_64/e/epel-release-7-5.noarch.rpm

Don't forget to configure ntp server on all nodes.

For all nodes:

yum install ceph-common ceph ceph-fuse ceph-deploy ntp -y  
systemctl enable ntpd
systemctl start ntpd

For thirst node:

cd /etc/ceph
ceph-deploy new node1 node2 node3
ceph-deploy mon create-initial

Check for running monitors:

ceph -s
  cluster 3a707493-a724-44f3-9ac6-9076b5d70a6c
   health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds
   monmap e1: 3 mons at {node1=192.168.6.214:6789/0,node2=192.168.6.215:6789/0,node3=192.168.6.216:6789/0}, election epoch 6, quorum 0,1,2 node1,node2,node3
   osdmap e1: 0 osds: 0 up, 0 in
    pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects
          0 kB used, 0 kB / 0 kB avail
               192 creating

For every node:

cd /etc/ceph
ceph-deploy gatherkeys nodeX
ceph-deploy disk zap nodeX:zvol/data/vdata
ceph-deploy osd prepare nodeX:zvol/data/vdata

where X is node number.

Check for all osd running:

  cluster 3a707493-a724-44f3-9ac6-9076b5d70a6c
   health HEALTH_WARN 982 pgs peering; 982 pgs stuck inactive; 985 pgs stuck unclean
   monmap e1: 3 mons at {node1=192.168.6.214:6789/0,node2=192.168.6.215:6789/0,node3=192.168.6.216:6789/0}, election epoch 6, quorum 0,1,2 node1,node2,node3
   osdmap e20: 3 osds: 3 up, 3 in
    pgmap v43: 1152 pgs, 3 pools, 0 bytes data, 0 objects
          115 MB used, 5082 GB / 5082 GB avail
                 3 active
               982 creating+peering
               167 active+clean

From any of nodes.

Create pools:

ceph osd pool delete rbd rbd --yes-i-really-really-mean-it
ceph osd pool create volumes 1024
ceph osd pool set volumes min_size 1
ceph osd pool set volumes size 2
ceph osd pool create images 1024
ceph osd pool set images min_size 1
ceph osd pool set images size 2
ceph osd pool create backups 1024
ceph osd pool set backups min_size 1
ceph osd pool set backups size 2
ceph osd pool create vms 1024
ceph osd pool set vms min_size 1
ceph osd pool set vms size 2

Setup cephx authentication:

ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images' >> /etc/ceph/ceph.client.cinder.keyring

copy /etc/ceph/ceph.client.cinder.keyring to other nodes.

yum install pacemaker corosync  resource-agents pcs -y

Set the same passsword for hacluster user on all nodes:

passwd hacluster

Enable and start pcsd service:

systemctl enable pcsd.service
systemctl start pcsd.service

Configure cluster from any of node:

pcs cluster auth node1 node2 node3
pcs cluster setup --name ovirt_alfa node1 node2 node3 --force
pcs cluster start --all

Check:

pcs status

Cluster name: ovirt_alfa
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Thu Apr 16 05:51:53 2015
Last change: Thu Apr 16 05:50:44 2015
Stack: corosync
Current DC: node1 (1) - partition with quorum
Version: 1.1.12-a14efad
3 Nodes configured
0 Resources configured

Online: [ node1 node2 node3 ]

Full list of resources:

PCSD Status:
node1: Online
node2: Online
node3: Online

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled

Setup properties:

pcs property set stonith-enabled=false
pcs property set no-quorum-policy=stop 

Create HA resource. Configure share IP address:

pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.6.240 cidr_netmask=24 op monitor interval=30s

If check is ok, ha cluster was successfully created.

On all nodes

Setup repo:

 cat << EOT > /etc/yum.repos.d/mariadb.repo
 [mariadb]
 name = MariaDB
 baseurl = http://yum.mariadb.org/10.0/centos7-amd64
 gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
 gpgcheck=1
 EOT

Installing:

 yum install MariaDB-Galera-server MariaDB-client rsync galera

start service:

 service mysql start
 chkconfig mysql on
 mysql_secure_installation

preparing for cluster:

 mysql -p
 GRANT USAGE ON *.* to sst_user@'%' IDENTIFIED BY 'PASS';
 GRANT ALL PRIVILEGES on *.* to sst_user@'%';
 FLUSH PRIVILEGES;
 exit
 service mysql stop

configuring cluster: (for nodeX)

 cat << EOT > /etc/my.cnf
 [mysqld]
 collation-server = utf8_general_ci
 init-connect = 'SET NAMES utf8'
 character-set-server = utf8
 binlog_format=ROW
 default-storage-engine=innodb
 innodb_autoinc_lock_mode=2
 innodb_locks_unsafe_for_binlog=1
 query_cache_size=0
 query_cache_type=0
 bind-address=0.0.0.0
 datadir=/var/lib/mysql
 innodb_log_file_size=100M
 innodb_file_per_table
 innodb_flush_log_at_trx_commit=2
 wsrep_provider=/usr/lib64/galera/libgalera_smm.so
 wsrep_cluster_address="gcomm://192.168.6.214,192.168.6.215,192.168.6.216"
 wsrep_cluster_name='scanex_galera_cluster'
 wsrep_node_address='192.168.6.X' # setup real node ip
 wsrep_node_name='nodeX' #  setup real node name
 wsrep_sst_method=rsync
 wsrep_sst_auth=sst_user:PASS
 EOT

(on node1)

 /etc/init.d/mysql start --wsrep-new-cluster

(on other nodes)

 /etc/init.d/mysql start

check on all nodes:

 mysql -p
 show status like 'wsrep%';

| Variable_name | Value | +——————————+————————————–+

+——————————+————————————–+

Remember, if all nodes will be down, actual node must be started with /etc/init.d/mysql start –wsrep-new-cluster. You should find an actual node. If you start node with not actual view, other nodes will issue error (see logs) - [ERROR] WSREP: gcs/src/gcs_group.cpp:void group_post_state_exchange(gcs_group_t*)():319: Reversing history: 0 → 0, this member has applied 140536161751824 more events than the primary component.Data loss is possible. Aborting.

On all nodes install rabbitmq:

yum install rabbitmq-server -y
systemctl enable rabbitmq-server

Configuring rabbitmq cluster. For first node:

systemctl start rabbitmq-server
systemctl stop rabbitmq-server

Copy .elang.cookie from node1 to other nodes:

scp /var/lib/rabbitmq/.erlang.cookie root@node2:/var/lib/rabbitmq/.erlang.cookie
scp /var/lib/rabbitmq/.erlang.cookie root@node3:/var/lib/rabbitmq/.erlang.cookie

On the target nodes ensure the correct owner, group, and permissions of the .erlang.cookie file:

chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
chmod 400 /var/lib/rabbitmq/.erlang.cookie

Start RabbitMQ on all nodes and verify the nodes are running:

systemctl start rabbitmq-server
rabbitmqctl cluster_status
Cluster status of node rabbit@nodeX ...
[{nodes,[{disc,[rabbit@nodeX]}]},
{running_nodes,[rabbit@nodeX]},
{cluster_name,<<"rabbit@nodeX">>},
{partitions,[]}]
...done.

Run the following commands on all nodes except the first one:

rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@node1
rabbitmqctl start_app

To verify the cluster status:

rabbitmqctl cluster_status
Cluster status of node rabbit@nodeX ...
[{nodes,[{disc,[rabbit@node1,rabbit@node2,rabbit@node3]}]},
{running_nodes,[rabbit@node3,rabbit@node2,rabbit@node1]},
{cluster_name,<<"rabbit@nodeX">>},
{partitions,[]}]
...done.

The message broker creates a default account that uses guest for the username and password. To simplify installation of your test environment, we recommend that you use this account, but change the password for it. Run the following command:
Replace RABBIT_PASS with a suitable password.

rabbitmqctl change_password guest RABBIT_PASS

To ensure that all queues, except those with auto-generated names, are mirrored across all running nodes it is necessary to set the policy key ha-mode to all. Run the following command on one of the nodes:

rabbitmqctl set_policy ha-all '^(?!amq\.).*' '{"ha-mode": "all"}'

Create the keystone database:

mysql -p
CREATE DATABASE keystone;
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'KEYSTONE_DBPASS';
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'KEYSTONE_DBPASS';

Enable openstack repository

yum install yum-plugin-priorities
rpm -ivh https://repos.fedorapeople.org/repos/openstack/openstack-juno/rdo-release-juno-1.noarch.rpm

Install components for all nodes:

yum install openstack-keystone python-keystoneclient -y

Generate a random value to use as the administration token during initial configuration (on one node):

openssl rand -hex 10

Edit the /etc/keystone/keystone.conf file and complete the following actions on all nodes.

In the [DEFAULT] section, define the value of the initial administration token:

Select Text
[DEFAULT]
...
admin_token = ADMIN_TOKEN
Replace ADMIN_TOKEN with the random value that you generated in a previous step.

In the [database] section, configure database access.

Select Text
[database]
...
connection = mysql://keystone:KEYSTONE_DBPASS@shareIP/keystone
Replace KEYSTONE_DBPASS with the password you chose for the database.

In the [token] section, configure the UUID token provider and SQL driver:

Select Text
[token]
...
provider = keystone.token.providers.uuid.Provider
driver = keystone.token.persistence.backends.sql.Token

In the [revoke] section, configure the SQL revocation driver:

Select Text
[revoke]
...
driver = keystone.contrib.revoke.backends.sql.Revoke

(Optional) To assist with troubleshooting, enable verbose logging in the [DEFAULT] section:

Select Text
[DEFAULT]
...
verbose = True

Create generic certificates and keys and restrict access to the associated files for node1:

keystone-manage pki_setup --keystone-user keystone --keystone-group keystone
chown -R keystone:keystone /var/log/keystone
chown -R keystone:keystone /etc/keystone/ssl
chmod -R o-rwx /etc/keystone/ssl

Then copy /etc/keystone/ssl with files to other nodes. After copy set permissions:

chown -R keystone:keystone /var/log/keystone
chown -R keystone:keystone /etc/keystone/ssl
chmod -R o-rwx /etc/keystone/ssl

Populate the Identity service database for node1:

su -s /bin/sh -c "keystone-manage db_sync" keystone

Check that database was populated:

mysql -p 
\r keystone
show tables;
+-----------------------+
| Tables_in_keystone    |
+-----------------------+
| assignment            |
| credential            |
| domain                |
| endpoint              |
| group                 |
| id_mapping            |
| migrate_version       |
| policy                |
| project               |
| region                |
| revocation_event      |
| role                  |
| service               |
| token                 |
| trust                 |
| trust_role            |
| user                  |
| user_group_membership |
+-----------------------+
18 rows in set (0.00 sec)

Copy /etc/keystone/keystone.conf to other nodes.

Try to start and stop Identity service on every of node. If service stared successfuly then start the Identity service as HA.

pcs resource create  ovirt_keystone systemd:openstack-keystone \
op monitor interval=30s timeout=20s \
op start timeout=120s \
op stop timeout=120s
pcs resource group add ovirt_gp  ClusterIP ovirt_keystone

Create tenants, users, and roles.
Configure the administration token and endpoint for every node:

export OS_SERVICE_TOKEN=6ccecc08c5f661c2bddd
export OS_SERVICE_ENDPOINT=http://shareIP:35357/v2.0

keystone tenant-create –name admin –description “Admin Tenant”

+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |           Admin Tenant           |
|   enabled   |               True               |
|      id     | 724adf6fdb324ba49385dba3a307d4d1 |
|     name    |              admin               |
+-------------+----------------------------------+  

keystone user-create --name admin --pass ADMIN_PASS --email test@test.ru
+----------+----------------------------------+
| Property |              Value               |
+----------+----------------------------------+
|  email   |           test@test.ru           |
| enabled  |               True               |
|    id    | d46ea880c23646b6858342eb98d32651 |
|   name   |              admin               |
| username |              admin               |
+----------+----------------------------------+

keystone role-create --name admin
+----------+----------------------------------+
| Property |              Value               |
+----------+----------------------------------+
|    id    | 81e202c2d28c41fdb1b6a45c772b4479 |
|   name   |              admin               |
+----------+----------------------------------+

keystone user-role-add --user admin --tenant admin --role admin
keystone tenant-create --name service --description "Service Tenant"
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |          Service Tenant          |
|   enabled   |               True               |
|      id     | 2250c77380534c3baaf5aa8cb1af0a05 |
|     name    |             service              |
+-------------+----------------------------------+

keystone service-create --name keystone --type identity --description "OpenStack Identity"
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |        OpenStack Identity        |
|   enabled   |               True               |
|      id     | 8e147209037a4071a8e04e0493c40a79 |
|     name    |             keystone             |
|     type    |             identity             |
+-------------+----------------------------------+

keystone endpoint-create \
--service-id $(keystone service-list | awk '/ identity / {print $2}') \
--publicurl http://shareIP:5000/v2.0 \
--internalurl http://shareIP:5000/v2.0 \
--adminurl http://shareIP:35357/v2.0 \
--region regionOne
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
|   adminurl  |    http://shareIP:35357/v2.0     |
|      id     | 66d2cc8c0f4a4f73ba083286d4704889 |
| internalurl |     http://shareIP:5000/v2.0     |
|  publicurl  |     http://shareIP:5000/v2.0     |
|    region   |            regionOne             |
|  service_id | 8e147209037a4071a8e04e0493c40a79 |
+-------------+----------------------------------+

Create OpenStack client environment scripts.
Create client environment scripts for the admin tenant and user. Future portions of this guide reference these scripts to load appropriate credentials for client operations.Edit the admin-openrc.sh file and add the following content:
export OS_TENANT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=ADMIN_PASS export OS_AUTH_URL=http://shareIP:35357/v2.0

Before you install and configure the Block Storage service, you must create a database, service credentials, and API endpoints. Create the cinder database and grant proper access to the cinder database:

CREATE DATABASE cinder;
GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'localhost' IDENTIFIED BY 'CINDER_DBPASS';
GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'CINDER_DBPASS';

To create the service credentials, complete these steps:

keystone user-create --name cinder --pass CINDER_PASS
+----------+----------------------------------+
| Property |              Value               |
+----------+----------------------------------+
|  email   |                                  |
| enabled  |               True               |
|    id    | 3fc1ae9c723143fba59ac7795ffc039c |
|   name   |              cinder              |
| username |              cinder              |
+----------+----------------------------------+

keystone user-role-add --user cinder --tenant service --role admin
keystone service-create --name cinder --type volume  --description "OpenStack Block Storage"
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |     OpenStack Block Storage      |
|   enabled   |               True               |
|      id     | 50c31611cc27447ea8c39fca8f7b86bf |
|     name    |              cinder              |
|     type    |              volume              |
+-------------+----------------------------------+

keystone service-create --name cinderv2 --type volumev2 --description "OpenStack Block Storage"
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |     OpenStack Block Storage      |
|   enabled   |               True               |
|      id     | 52002528a42b41ffa2abf8b8bb799f08 |
|     name    |             cinderv2             |
|     type    |             volumev2             |
+-------------+----------------------------------+

keystone endpoint-create \
--service-id $(keystone service-list | awk '/ volumev2 / {print $2}') \
--publicurl http://shareIP:8776/v2/%\(tenant_id\)s \
--internalurl http://shareIP:8776/v2/%\(tenant_id\)s \
--adminurl http://shareIP:8776/v2/%\(tenant_id\)s \
--region regionOne
+-------------+--------------------------------------+
|   Property  |                Value                 |
+-------------+--------------------------------------+
|   adminurl  | http://shareIP:8776/v1/%(tenant_id)s |
|      id     |   140f2f2a3f7c4f31b7c3308660c1fe31   |
| internalurl | http://shareIP:8776/v1/%(tenant_id)s |
|  publicurl  | http://shareIP:8776/v1/%(tenant_id)s |
|    region   |              regionOne               |
|  service_id |   50c31611cc27447ea8c39fca8f7b86bf   |
+-------------+--------------------------------------+

keystone endpoint-create \
--service-id $(keystone service-list | awk '/ volumev2 / {print $2}') \
--publicurl http://shareIP:8776/v2/%\(tenant_id\)s \
--internalurl http://shareIP:8776/v2/%\(tenant_id\)s \
--adminurl http://shareIP:8776/v2/%\(tenant_id\)s \
--region regionOne
+-------------+--------------------------------------+
|   Property  |                Value                 |
+-------------+--------------------------------------+
|   adminurl  | http://shareIP:8776/v2/%(tenant_id)s |
|      id     |   233dcfa7ff6048a4ad72bec9c962df60   |
| internalurl | http://shareIP:8776/v2/%(tenant_id)s |
|  publicurl  | http://shareIP:8776/v2/%(tenant_id)s |
|    region   |              regionOne               |
|  service_id |   52002528a42b41ffa2abf8b8bb799f08   |
+-------------+--------------------------------------+

Install and configure Block Storage controller components on all nodes:

yum install openstack-cinder python-cinderclient python-oslo-db targetcli python-oslo-db MySQL-python -y

Edit the /etc/cinder/cinder.conf file and complete the following actions for all nodes.

In the [database] section, configure database access:

Select Text
[database]
...
connection = mysql://cinder:CINDER_DBPASS@shareIP/cinder
Replace CINDER_DBPASS with the password you chose for the Block Storage database.

In the [DEFAULT] section, configure RabbitMQ message broker access:

Select Text
[DEFAULT]
...
rpc_backend = rabbit
rabbit_host = node1:5672,node2:5672,node3:5672
rabbit_password = RABBIT_PASS
rabbit_retry_interval=1
rabbit_retry_backoff=2
rabbit_max_retries=0
rabbit_durable_queues=true
rabbit_ha_queues=true
Replace RABBIT_PASS with the password you chose for the guest account in RabbitMQ.

In the [DEFAULT] and [keystone_authtoken] sections, configure Identity service access:

Select Text
[DEFAULT]
...
auth_strategy = keystone
[keystone_authtoken]
...
auth_uri = http://shareIP:5000/v2.0
identity_uri = http://shareIP:35357
admin_tenant_name = service
admin_user = cinder
admin_password = CINDER_PASS
Replace CINDER_PASS with the password you chose for the cinder user in the Identity service.

For rbd support:

rbd_pool=volumes  
# The RADOS client name for accessing rbd volumes - only set
rbd_user=cinder # we setup cephx for client.cinder. See 1.5.
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot=false
# The libvirt uuid of the secret for the rbd_user volumes. Generate with use uuidgen
rbd_secret_uuid=d668041f-d3c1-417b-b587-48261ad2abf3
rbd_max_clone_depth=5
volume_driver=cinder.volume.drivers.rbd.RBDDriver

[Note] Note Comment out any auth_host, auth_port, and auth_protocol options because the identity_uri option replaces them.

In the [DEFAULT] section, configure the my_ip option to use the management interface IP address of the controller node:

Select Text
[DEFAULT]
...
my_ip = 192.168.6.240 #shareIP

(Optional) To assist with troubleshooting, enable verbose logging in the [DEFAULT] section:

Select Text
[DEFAULT]
...
verbose = True

Populate the Block Storage database:

su -s /bin/sh -c "cinder-manage db sync" cinder

2015-04-16 09:58:32.119 26968 INFO migrate.versioning.api [-] 0 -> 1...
2015-04-16 09:58:33.761 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:33.762 26968 INFO migrate.versioning.api [-] 1 -> 2...
2015-04-16 09:58:34.303 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:34.304 26968 INFO migrate.versioning.api [-] 2 -> 3...
2015-04-16 09:58:34.470 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:34.470 26968 INFO migrate.versioning.api [-] 3 -> 4...
2015-04-16 09:58:35.167 26968 INFO 004_volume_type_to_uuid [-] Created foreign key volume_type_extra_specs_ibfk_1
2015-04-16 09:58:35.173 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:35.173 26968 INFO migrate.versioning.api [-] 4 -> 5...
2015-04-16 09:58:35.308 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:35.309 26968 INFO migrate.versioning.api [-] 5 -> 6...
2015-04-16 09:58:35.492 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:35.492 26968 INFO migrate.versioning.api [-] 6 -> 7...
2015-04-16 09:58:35.666 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:35.666 26968 INFO migrate.versioning.api [-] 7 -> 8...
2015-04-16 09:58:35.757 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:35.757 26968 INFO migrate.versioning.api [-] 8 -> 9...
2015-04-16 09:58:35.857 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:35.857 26968 INFO migrate.versioning.api [-] 9 -> 10...
2015-04-16 09:58:35.957 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:35.958 26968 INFO migrate.versioning.api [-] 10 -> 11...
2015-04-16 09:58:36.117 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:36.119 26968 INFO migrate.versioning.api [-] 11 -> 12...
2015-04-16 09:58:36.253 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:36.253 26968 INFO migrate.versioning.api [-] 12 -> 13...
2015-04-16 09:58:36.419 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:36.419 26968 INFO migrate.versioning.api [-] 13 -> 14...
2015-04-16 09:58:36.544 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:36.545 26968 INFO migrate.versioning.api [-] 14 -> 15...
2015-04-16 09:58:36.600 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:36.601 26968 INFO migrate.versioning.api [-] 15 -> 16...
2015-04-16 09:58:36.751 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:36.751 26968 INFO migrate.versioning.api [-] 16 -> 17...
2015-04-16 09:58:37.294 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:37.294 26968 INFO migrate.versioning.api [-] 17 -> 18...
2015-04-16 09:58:37.773 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:37.773 26968 INFO migrate.versioning.api [-] 18 -> 19...
2015-04-16 09:58:37.896 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:37.896 26968 INFO migrate.versioning.api [-] 19 -> 20...
2015-04-16 09:58:38.030 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:38.030 26968 INFO migrate.versioning.api [-] 20 -> 21...
2015-04-16 09:58:38.053 26968 INFO 021_add_default_quota_class [-] Added default quota class data into the DB.
2015-04-16 09:58:38.058 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:38.058 26968 INFO migrate.versioning.api [-] 21 -> 22...
2015-04-16 09:58:38.173 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:38.174 26968 INFO migrate.versioning.api [-] 22 -> 23...
2015-04-16 09:58:38.263 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:38.264 26968 INFO migrate.versioning.api [-] 23 -> 24...
2015-04-16 09:58:38.642 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:38.642 26968 INFO migrate.versioning.api [-] 24 -> 25...
2015-04-16 09:58:39.443 26968 INFO migrate.versioning.api [-] done
2015-04-16 09:58:39.444 26968 INFO migrate.versioning.api [-] 25 -> 26...
2015-04-16 09:58:39.455 26968 INFO 026_add_consistencygroup_quota_class [-] Added default consistencygroups quota class data into the DB.
2015-04-16 09:58:39.461 26968 INFO migrate.versioning.api [-] done

Try to start

systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service openstack-cinder-volume.service

check logs:

tail -f /var/log/cinder/api.log /var/log/cinder/scheduler.log
==> /var/log/cinder/api.log <==
2015-04-16 10:34:30.442 30784 INFO cinder.openstack.common.service [-] Started child 30818
2015-04-16 10:34:30.443 30818 INFO eventlet.wsgi.server [-] (30818) wsgi starting up on http://0.0.0.0:8776/
2015-04-16 10:34:30.445 30784 INFO cinder.openstack.common.service [-] Started child 30819
2015-04-16 10:34:30.447 30819 INFO eventlet.wsgi.server [-] (30819) wsgi starting up on http://0.0.0.0:8776/
2015-04-16 10:34:30.448 30784 INFO cinder.openstack.common.service [-] Started child 30820
2015-04-16 10:34:30.449 30820 INFO eventlet.wsgi.server [-] (30820) wsgi starting up on http://0.0.0.0:8776/
2015-04-16 10:34:30.450 30784 INFO cinder.openstack.common.service [-] Started child 30821
2015-04-16 10:34:30.452 30821 INFO eventlet.wsgi.server [-] (30821) wsgi starting up on http://0.0.0.0:8776/
2015-04-16 10:34:30.453 30784 INFO cinder.openstack.common.service [-] Started child 30822
2015-04-16 10:34:30.454 30822 INFO eventlet.wsgi.server [-] (30822) wsgi starting up on http://0.0.0.0:8776/

==> /var/log/cinder/scheduler.log <==
2015-04-16 10:34:29.940 30785 INFO oslo.messaging._drivers.impl_rabbit [req-80c27067-712a-47c3-a8e3-a448e1677e8a - - - - -] Connecting to AMQP server on node3:5672
2015-04-16 10:34:29.961 30785 INFO oslo.messaging._drivers.impl_rabbit [req-80c27067-712a-47c3-a8e3-a448e1677e8a - - - - -] Connected to AMQP server on node3:5672
2015-04-16 10:34:30.310 30785 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on node3:5672
2015-04-16 10:34:30.322 30785 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on node3:5672
2015-04-16 10:37:14.587 30785 INFO cinder.openstack.common.service [-] Caught SIGTERM, exiting
2015-04-16 10:37:43.146 31144 INFO cinder.service [-] Starting cinder-scheduler node (version 2014.2.2)
2015-04-16 10:37:43.148 31144 INFO oslo.messaging._drivers.impl_rabbit [req-6e41c1e3-1ffe-4918-bdb2-b99e665713a6 - - - - -] Connecting to AMQP server on node2:5672
2015-04-16 10:37:43.167 31144 INFO oslo.messaging._drivers.impl_rabbit [req-6e41c1e3-1ffe-4918-bdb2-b99e665713a6 - - - - -] Connected to AMQP server on node2:5672
2015-04-16 10:37:43.517 31144 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on node1:5672
2015-04-16 10:37:43.530 31144 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on node1:5672

check service:

source admin-openrc.sh
cinder service-list
+------------------+-------+------+---------+-------+----------------------------+-----------------+
|      Binary      |  Host | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+-------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | imm14 | nova | enabled |   up  | 2015-04-16T14:52:09.000000 |       None      |
+------------------+-------+------+---------+-------+----------------------------+-----------------+
|  cinder-volume   | imm14 | nova | enabled |   up  | 2015-04-16T15:13:37.000000 |       None      |
+------------------+-------+------+---------+-------+----------------------------+-----------------+

Create volume

cinder create --display-name demo-volume2 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2015-04-16T15:41:01.202976      |
| display_description |                 None                 |
|     display_name    |             demo-volume2             |
|      encrypted      |                False                 |
|          id         | 5c7d35e8-ea55-4aa9-8a57-bb0de94552e7 |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

Check for image creation

cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 5c7d35e8-ea55-4aa9-8a57-bb0de94552e7 | available | demo-volume2 |  1   |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

rbd -p volumes ls -l
NAME                                         SIZE PARENT FMT PROT LOCK
volume-5c7d35e8-ea55-4aa9-8a57-bb0de94552e7 1024M          2

Stop service and start as HA

systemctl stop openstack-cinder-api.service openstack-cinder-scheduler.service openstack-cinder-volume.service

pcs resource create  ovirt_cinder_api systemd:openstack-cinder-api \
op monitor interval=30s timeout=20s \
op start timeout=120s \
op stop timeout=120s

pcs resource create  ovirt_cinder_scheduler systemd:openstack-cinder-scheduler \
op monitor interval=30s timeout=20s \
op start timeout=120s \
op stop timeout=120s

pcs resource create  ovirt_cinder_volume systemd:openstack-cinder-volume \
op monitor interval=30s timeout=20s \
op start timeout=120s \
op stop timeout=120s
pcs resource group add ovirt_gp ovirt_cinder_api ovirt_cinder_scheduler ovirt_cinder_volume

Profile of the author