Configuring HA NFS export of home directories stored on ceph file storage and mounting on clients with automount within FreeIPA infrastructure

Recently i faced with task of creating HA NFS export of home directories of cephfs storage within FreeIPA infrastructure.

All devices joined to domen example.com. Network is 192.168.10.0/24. Cephfs storage consists of 3 nodes with cephfs mounted /cephfs directory. All home directories of users are stored in /cephfs/home.

Home directories must be mount by using automount because all users have sudo permissions on their own PCs. All OSs are Centos 7.

This article will only focused on configuring of NFS HA and automount within an FreeIPA domain.

Configuring HA NFS export required to install temporary server with name nfs-home-server.example.com (IP - 192.168.10.100). Server must be joined to domain example.com.
The server that has been joined can be poweroff forever.

On directory server ipa-server.example.com.
Creating nfs service principal:

kinit admin #can be used any user with admin permissions
ipa service-add nfs/nfs-home-server.example.com
ipa service-add nfs/client.example.com # must be created for every server or that will be used automount

Creating automount record:

ipa automountmap-add default auto.home
ipa automountkey-add default --key "/home" --info auto.home auto.master
ipa automountkey-add default --key "*" --info "-fstype=nfs4,rw,sec=krb5,soft,rsize=8192,wsize=8192 nfs-home-server.example.com:/cephfs/home/&" auto.home

For every of cephfs nodes.
Installing:

yum install rpcbind nfs-utils -y

Configuring export:

echo "/cephfs/home  *(rw,sec=sys:krb5:krb5i:krb5p)" > /etc/exports

Edit /etc/sysconfig/nfs

SECURE_NFS=yes

Saving Kerberos keytabs:

kinit admin #can be used any user with admin permissions
ipa-getkeytab -s ipa-server.example.com -p host/nfs-home-server.example.com -k /etc/krb5.keytab
ipa-getkeytab -s ipa-server.example.com -p nfs/nfs-home-server.example.com -k /etc/krb5.keytab

Starting nfs server:

systemctl enable rpcbind nfs-server nfs-lock nfs-idmap
systemctl start rpcbind nfs-server nfs-lock nfs-idmap

Creating HA shared IP - 192.168.10.100.
Instaling pcs cluster on all nodes:

yum install pacemaker corosync  resource-agents pcs -y

Configuring pcs cluster
Set the same passsword for hacluster user on all nodes:

passwd hacluster

Enable and start pcsd service:

systemctl enable pcsd.service
systemctl start pcsd.service

Configure cluster from any of node:

pcs cluster auth ceph-node1.example.com ceph-node2.example.com ceph-node3.example.com
pcs cluster setup --name nfs-ha  ceph-node1.example.com ceph-node2.example.com ceph-node3.example.com --force
pcs cluster start --all

Check:

pcs status

Setup properties:

pcs property set stonith-enabled=false
pcs property set no-quorum-policy=stop 

Creating HA resource, configuring share IP address:

pcs resource create NFS_Shared_IP ocf:heartbeat:IPaddr2 ip=192.168.10.100 cidr_netmask=24 op monitor interval=30s

If check is ok, ha cluster was successfully created.

Saving Kerberos keytab:

kinit admin #can be used any user with admin permissions
ipa-getkeytab -s ipa-server.example.com -p nfs/client.example.com -k /etc/krb5.keytab

Installing , configuring and starting automount:

yum install autofs -y
ipa-client-automount --location=default
systemctl start autofs

Profile of the author