Cluster 20. Install & setup environment needed for clustered virtualization

KVM Installation & initial setup

Install packages needed for KVM:
yum install –y kvm virt-manager virt-install libvirt libvirt-python libguestfs-tools syslinux pciutils
Verify that the packages were installed correctly:
lsmod | grep kvm # to see if kvm anf kvm_intel modules are loaded

Packages descripted:

• kvm - hypervisor
• virt-manager - package contains several command-line utilities (also GUI tools) for building and installing new virtual machines, and virt-clone for cloning existing virtual machines
• libvirt - is a C toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes). The library aims at providing a long term stable C API for different virtualization mechanisms. It currently supports QEMU, KVM, XEN, OpenVZ, LXC, and VirtualBox.
• libvirt-python - package provides a module that permits applications written in the Python programming language to call the interface supplied by the libvirt library, to manage the virtualization capabilities of recent versions of Linux (and other OSes).
• libguestfs-tool - This package contains the guestfish (shell and command-line tool for examining and modifying virtual machine filesystems) and various virtualization tools, including virt-cat, virt-df, virt-edit, virt-filesystems, virt-inspector, virt-ls, virt-make-fs, virt-rescue, virt-resize, virt-tar, and virt-win-reg
• syslinux - is a suite of bootloaders, currently supporting DOS FAT filesystems, Linux ext2/ext3 filesystems (EXTLINUX), PXE network boots (PXELINUX), or ISO 9660 CD-ROMs  (ISOLINUX). It also includes a tool, MEMDISK, which loads legacy operating systems from these media.
• pciutils - The pciutils package contains various utilities for : inspecting and setting devices connected to the PCI bus.

Check and destroy the default libvirtd bridge (By default, VMs will only have network access to other VMs on the same server (and to the host itself)

systemctl start libvirtd
systemctl status libvirtd

via private network 192.168.122.0. If you want the VMs to have access to your LAN, then you must create a network bridge on the host.):

ip route | grep virbr0

virsh net-destroy default 

virsh net-autostart default --disable 

virsh net-undefine default

ip route | grep virbr0

Check and disable libvirtd:
systemctl status libvirtd
systemctl stop libvirtd
systemctl disable libvirtd

Provision Planning

The servers I'm using to write this tutorial are a little modest in the RAM department with only 16 GiB of RAM. We need to subtract at least 2 GiB for the host nodes, leaving us with a total of 14 GiB. That needs to be divided up among all your servers. Now, nothing says you have to use it all, of course. It's perfectly fine to leave some RAM unallocated for future use. This is really up to you and your needs.

Let's put together a table with the RAM we plan to allocate and summarizing the LV we're going to create for each server. The LV will be named after the server they'll be assigned to with the suffix _0. Later, if we add a second "hard drive" to a server, it will have the suffix _1 and so on.

Server	RAM (GiB)	Storage Pool (VG)	LV name	LV size
vm01-nagios	2	agrp-c01n01	vm01-nagios_0	150 GB
vm02-www	4	agrp-c01n01	vm02-www_0	150 GB
vm03-mysql	3	agrp-c01n02	vm03-mysql_0	100 GB
vm04-asterisk	4	agrp-c01n02	vm04-asterisk_0	100 GB
Total	13 GiB	--------------	---------------	500 GB

As you see, we'll use 13 GiB of RAM, so remaining RAM amount will be 3 GiB (16-13=3). And we'll use 500GB of storage, so remaining amount of VM dedicated storage (both DRBD r0+r1=1000GB) will be 500GB (1000-500=500)
The same approach can be used for CPU - read this blog-post - how-many-vCPU-per-pCPU

Provision Shared CentOS ISOs

Before we can install the OS, we need to copy the installation media and our driver disk, if needed, and put them in the /shared/files.

For our needs we'll install CentOS6 & CentOS7 machines (for Windows machines, please visit: AN!Cluster_Tutorial - alteeve.com). So download both CentOS 6 & 7 Minimal images from one of the nodes and then send it to the other (I'll be using one of our office machines):

pcs cluster start --all # if didn't start previously

rsync -av --progress CentOS-7-x86_64-Minimal-1708.iso root@172.16.51.1:/shared/files/
rsync -av --progress CentOS-6.9-x86_64-minimal.iso root@172.16.3.235:/shared/files

Creating Storage for VMs

Earlier, we used parted to examine our free space and create our DRBD partitions. Unfortunately, parted shows sizes in GB (base 10) where LVM uses GiB (base 2). If we used LVM's "xxG size notation, it will use more space than we expect, relative to our planning in the parted stage. LVM doesn't allow specifying new LV sizes in GB instead of GiB, so here we will specify sizes in MiB to help narrow the differences.

Storage creating is the same for all VM. So I'll show only one LV creation:

lvcreate -L 150000M -n vm01-nagios_0 agrp-c01n01_vg0
or you can use bytes-count (ie 150GB=150*1024*1024*1024bytes=161061273600b):
lvcreate -L 161061273600b -n vm01-nagios_0 agrp-c01n01_vg0

lvdisplay /dev/agrp-c01n01_vg0/vm01-nagios_0
To remove lv:
lvremove /dev/agrp-c01n01_vg0/vm01-nagios_0 

Creating OpenVSwitch group for VMs

Find name of the bridge:

ovs-vsctl list Bridge | grep name

Add port group  to the file /shared/provision/ovs-network.xml (if more than one vlan is needed – add <portgroup>..</portgroup> for every vlan)

<network>
<name>ovs-network</name>
<forward mode='bridge'/>
<bridge name='ovs_kvm_bridge'/>
<virtualport type='openvswitch'/>
    <portgroup name='vlan-51'>
         <vlan>
            <tag id='51'/>
        </vlan>
   </portgroup>
</network>

To add networ to a KVM (from both nodes): 

systemctl start libvirtd
virsh net-define /shared/provision/ovs-network.xml
net-list --all
virsh net-start ovs-network
virsh net-autostart ovs-network 

virsh net-list

systemctl stop libvirtd

To delete network from KVM (if needed - from both nodes):
virsh net-list
virsh net-destroy ovs-network
virsh net-autostart --disable ovs-network
virsh net-undefine ovs-network

Virtio

So-called "full virtualization" is a nice feature because it allows you to run any operating system virtualized. However, it's slow because the hypervisor has to emulate actual physical devices such as RTL8139 network cards . This emulation is both complicated and inefficient. 

Virtio is a virtualization standard for network and disk device drivers where just the guest's device driver "knows" it is running in a virtual environment, and cooperates with the hypervisor. This enables guests to get high performance network and disk operations, and gives most of the performance benefits of paravirtualization.

Creating virt-install call

touch /shared/provision/vm01-nagios.sh 

chmod 755 /shared/provision/vm01-nagios.sh 

vim /shared/provision/vm01-nagios.sh
virt-install --connect qemu:///system \
--name=vm01-nagios \
--ram=2048 \
--arch=x86_64 \
--vcpus=2 \
--location=/shared/files/CentOS-6.9-x86_64-minimal.iso \
--os-variant=centos6.9 \
--network network=ovs-network,portgroup=vlan-51,model=virtio \
--disk path=/dev/agrp-c01n01_vg0/vm01-nagios_0,bus=virtio \
--graphics none \
--extra-args 'console=ttyS0'

Options Described:

1. --connect qemu:///system - This tells virt-install to use the QEMU hardware emulator (as opposed to Xen, for example) and to install the server on to local node.
2. --name vm01-nagios - This sets the name of the server. It is the name we will use in the cluster configuration and whenever we use the libvirtd tools, like virsh.
3. --ram 2048 - This sets the amount of RAM, in MiB, to allocate to this server. Here, we're allocating 2 GiB, which is 2048 MiB.
4. --arch x86_64 - i386 – 32bit old CPUs, i686 – 32bit new CPUs, x86-64 – 64bit CPUs
5. --vcpus 2 - This sets the number of CPU cores to allocate to this server. Here, we're allocating two CPUs.
6. --location /shared/files/CentOS-6.9-x86_64-minimal.iso - Distribution tree installation source. virt-install can recognize certain distribution trees and fetches a bootable kernel/initrd pair to launch the install.
7. --os-variant centos6.9 - This tweaks the virt-manager's initial method of running and tunes the hypervisor to try and get the best performance for the server. There are many possible values here for many, many different operating systems. If you run osinfo-query os on your node, you will get a full list of available operating systems. If you can't find your exact operating system, select the one that is the closest match.
8. --network network=ovs-network,portgroup=vlan-51,model=virtio - This tells the hypervisor that we want to create a network card using the virtio "hardware" and that we want it plugged into the ovs-network bridge's  vlan-51 portgroup. We only need one network card, but if you wanted two or more, simply repeat this command. If you create two or more bridges, you can have different network devices connect to different bridges.
9. --disk path=/dev/agrp-c01n01_vg0/vm01-nagios_0,bus=virtio - This tells the hypervisor what LV to use for the server's "hard drive". It also tells it to use the virtio emulated SCSI controller.
10.  --graphics none - we'll use only CLI without any GUI (also for installation)
11. --extra-args 'console=ttyS0' - this is needed to see installation process from console

Installing VM on the node

We can install any server from either node. However, we know that each server has a preferred node, so it's sensible to use that host for the installation stage. In this case of vm01-nagios, the preferred host is agrp-c01n01, so we'lluse it to start the installation.




ssh to the agrp-c01n01
systemctl start libvirtd
/shared/provision/vm01-nagios.sh
Go through steps of text-mode installation
To exit installed VM hit Ctrl+5 (remote connect) or Ctrl+] (local connect)
To connect to the installed VM virsh console vm01-nagios
To list installed systems and their operating mode virsh list --all
To start "shut-offed" VM virsh start vm01-nagios


Steps to perform on a VM after installation (if you need them): 







For CentOS6:


chkconfig ip6tables off
service ip6tables stop
cat /etc/sysconfig/network

NETWORKING=yes
NETWORKING_IPV6=no

vi /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
kernel.panic = 5 # self-reboot in 5 seconds when panicking 

sysctl -p
vi /etc/sysconfig/network-scripts/ifcfg-eth0

NM_CONTROLLED=no
ONBOOT=yes

service network restart
ip route


For CentOS7 (this version by default does self-restart on kernel panicking):

1. systemctl stop NetworkManager
2. systemctl disable NetworkManager
3. chkconfig network on
4. systemctl start network
5. vi /etc/sysconfig/network-scripts/ifcfg-eth0
1. ONBOOT=yes
6. systemctl restart network
7. ip route

To learn IP addresses and OVS port names of the VM (execute from the node where VM is situated): 
for name in $(virsh list | awk '{print $2}' | grep -v '^$\|Name'); do echo $name;virsh domiflist $name; echo""; done 

vm02-www 
Interface Type     Source           Model   MAC 
---------------------------------------------------------------------- 
vnet0      bridge   ovs-network   virtio    52:54:00:77:3a:a0

nagios
Interface Type     Source           Model   MAC
----------------------------------------------------------------------
vnet1      bridge   ovs-network   virtio    52:54:00:77:3d:19

VM shutdown test

To test if node can be shutdown:

virsh shutdown vm01-nagios

If shutdown is not performed and node remains active (mostly this is problem on CentOS6):

virsh console vm01-nagios

yum -y install acpid

service acpid start

chkconfig --level 235 acpid on
chkconfig --list acpid

Test again:

virsh shutdown vm02-www

ACPI (Advanced Configuration and Power Interface) — enhanced interface for power supply management. ACPI is the component of many modern computers. It gives PC users ability to manage power supply programmatically and also query batteries state and parameters.

This tutorials were used to understand and setup clustering: 

AN!Cluster
clusterlabs.org
redhat.com

Cisco ASA how to find why packet is not going in or out through VPN

For example we want to check access through INSIDE interface from 10.10.100.100 outside client's tcp port 30000 to the 10.20.100.100 internal server's 3389 tcp port (Windown RDP).

First you need to check packet "movement":

packet-tracer input INSIDE tcp 10.10.100.100 30000 10.20.100.100 3389 detailed

Correct all problems appeared in each Phase. If the only "Drop" result is on VPN Phase, then:

sh run route | grep 10.20.100.100 # found gateway is 10.30.100.100

sh run group-policy | grep 10.30.100.100 # found group-policy name is GP_10.30.100.100

sh run group-policy GP_110.30.100.100 | grep vpn-filter # found ACL name is INSIDE.30.100.100.vpn.filter

No you can verify this ACL and add needed permissions

Install Gnome Desktop GUI to the CentOS7

1. yum install yum-utils
2. yum grouplist | grep -i desktop
3. yum groupinstall "Gnome Desktop"
4. systemctl set-default graphical.target
5. reboot

Asterisk retransmission timeout

Sometimes Asterisk log show this message:

WARNING[9380] chan_sip.c: Retransmission timeout reached on transmission 1a1df1ca5a8cc898668aaabf73852da2@10.10.10.10:5060 for seqno 102 (Critical Request) -- See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions

This can be because of several reasons and usually the easiest way to get rid of this is to:

core restart now from Asterisk console.

While this solves the problem, you can have many calls you don't want to lose, in such a situation you can do:

core restart gracefully 

now new calls will not come into system, and restart will be done as long as all (old) calls are ended.

But I found one way around - to issue core restart gracefully - when you have retransmission error and execute - core abort shutdown - when error disappears. This approach gives you ability to accept phone calls and also eliminates problems caused by retransmission error (bad voice, lost calls, simultaneous calls to the same operator etc.).

Save below script into the file (i.e. named: retransmission_timeout_resolving.py) and setup cron to execute this file every minute ( * * * * * /root/retransmission_timeout_resolving.py):

#!/usr/bin/python

from subprocess import PIPE,Popen

import time

try:

 proc1 = Popen(["/usr/bin/tail", "-n", "10", "/var/log/asterisk/messages"], stdout=PIPE)

 tail = (proc1.communicate()[0].split("\n"))

except:

 LOG_LINE = "77 ==> " + str(time.ctime()) + "\n"

 with open('/var/log/asterisk/RETRANSMISSION_ERR.log', 'a') as f:

  f.write(LOG_LINE)

for x in tail:

 if 'SIP+Retransmissions' in x:

  ERR = 1

  break

 elif 'SIP+Retransmissions' not in x:

  ERR = 0

if ERR == 1:

 proc2 = Popen(["/usr/sbin/asterisk", "-rx", "core restart gracefully"])

elif ERR == 0:

 proc3 = Popen(["/usr/sbin/asterisk", "-rx", "core abort shutdown"])

else:

 ERR = "SMTH WRONG HAPPENED"

LOG_LINE = str(ERR) + "  ==> " + str(time.ctime()) + "\n"

with open('/var/log/asterisk/RETRANSMISSION_ERR.log', 'a') as f:

 f.write(LOG_LINE)

Cluster 19. Fencing level 2 with SNMP.

Please, reread these blog-posts => Cluster 7   / Cluster 11 / Cluster 12

We have level-1 fencing (iLO/IPMI-fencing) setup. But what is the problem with this? The problem is when a node vanishes and fencing fails (if one node loses power - the IPMI fencing will fail for that node). Then, not knowing what the other node might be doing, the only safe option is to block, otherwise you risk a split-brain. To verify that:

1. pcs cluster start --all
2. then pull power cable out of one of the nodes
3. dlm_tool ls # you'll see that:
1. new change    member 1 joined 0 remove 1 failed 1 seq 2,2
2. new status    wait fencing

This means that dlm will wait till the powered-off node status get cleared. You'll see the same messages for both "shared" and "clvmd" lock-spaces. This means that GFS2 file-system and CLVMD will hung (pcs status will still show this resources as "Started"). This is because  GFS2 file systems will freeze to ensure data integrity in the event of a failed fence. 

This is why multiple-level fencing is so important.

Overview SNMP fencing

The logic behind this mechanism is very simple: once a node has been marked as dead the agent uses the SNMP SET method to tell the managed switch to shut the ports down. For SNMP fencing we'll use fence_ifmib (see: pcs stonith list fence_ifmib and pcs stonith describe fence_ifmib). Only two OIDs are needed by the agent: ifDescr and ifAdminStatus. The first is used to match the interface name used on the Cisco device (fence_ifmib can be used with any vendor while it's supporting SNMP) with the one provided in the cluster configuration, the latter to get/set the port status.

Setup Cisco stack switches to support SNMP

As we setup in Cluster 3, Switch IP for the 1st cluster is 10.10.53.12 and the 1st cluster uses ports gi1/0/1-4,17 & gi2/0/1-4,17. So:

1. agrp-c01n01 uses ports
1. 1/0/1, 1/0/3, 2/0/2, 2/0/4
2. (config)# int ra gi1/0/1, gi1/0/3, gi2/0/2, gi2/0/4
3. (config-if-range) # description agrp-c01n01
4. these are all members of the channel-group 2
6. (config)#int Port-channel 2
7. (config-if)#description agrp-c01n01
2. agrp-c01n02 uses ports
1. 2/0/1, 2/0/3, 1/0/2, 1/0/4
2. (config)# int ra gi2/0/1, gi2/0/3, gi1/0/2, gi1/0/4
3. (config-if-range) # description agrp-c01n02
4. these are all members of the channel-group 3
6. (config)#int Port-channel 3
7. (config-if)#description agrp-c01n02
3. save and verify with show interface status | incl agrp-c01

Setup ACL for SNMP view:

ip access-list standard agrp-c01-acl

  permit 10.10.53.1

  permit 10.10.53.2

  deny any

Setup SNMP view and community for that view:

Setup community (this enables SNMP agent):

snmp-server community agrp-c01-community RW agrp-c01-acl

Test with:

snmpwalk -v 2c -c agrp-c01-community 10.10.53.12 # huge list of OID will be shown

Find IfIndex of needed interfaces (we'll use them to restrict access in set up community to the needed values only):
You can search only for Port-channel interfaces if you use LACP:
agrp-c01n01: show snmp mib ifmib ifindex | incl Port-channel2
agrp-c01n02: show snmp mib ifmib ifindex | incl Port-channel3
Or search for all connected interfaces, if don't using LACP:

agrp-c01n01: show snmp mib ifmib ifindex | incl net(1/0/[13]|2/0/[24]):

agrp-c01n02: show snmp mib ifmib ifindex | incl net(2/0/[13]|1/0/[24]):

Setup SNMP view:
If you use LACP:
#agrp-c01n01
#For Port-channel2: Ifindex = 5002

snmp-server view agrp-c01-view ifDescr.5002 included 

snmp-server view agrp-c01-view ifAdminStatus.5002 included 

#agrp-c01n02
#For Port-channel3: Ifindex = 5003

snmp-server view agrp-c01-view ifDescr.5003 included 

snmp-server view agrp-c01-view ifAdminStatus.5003 included 

If you don't use LACP:
#agrp-c01n01

#For GigabitEthernet2/0/4: Ifindex = 10604

snmp-server view agrp-c01-view ifDescr.10604 included 

snmp-server view agrp-c01-view ifAdminStatus.10604 included 

#For GigabitEthernet2/0/2: Ifindex = 10602

snmp-server view agrp-c01-view ifDescr.10602 included 

snmp-server view agrp-c01-view ifAdminStatus.10602 included 

#For GigabitEthernet1/0/3: Ifindex = 10103

snmp-server view agrp-c01-view ifDescr.10103 included 

snmp-server view agrp-c01-view ifAdminStatus.10103 included 

#For GigabitEthernet1/0/1: Ifindex = 10101

snmp-server view agrp-c01-view ifDescr.10101 included 

snmp-server view agrp-c01-view ifAdminStatus.10101 included 

#agrp-c01n02

#For GigabitEthernet2/0/3: Ifindex = 10603

snmp-server view agrp-c01-view ifDescr.10603 included 

snmp-server view agrp-c01-view ifAdminStatus.10603 included 

#For GigabitEthernet2/0/1: Ifindex = 10601

snmp-server view agrp-c01-view ifDescr.10601 included 

snmp-server view agrp-c01-view ifAdminStatus.10601 included 

#For GigabitEthernet1/0/4: Ifindex = 10104

snmp-server view agrp-c01-view ifDescr.10104 included 

snmp-server view agrp-c01-view ifAdminStatus.10104 included 

#For GigabitEthernet1/0/2: Ifindex = 10102

snmp-server view agrp-c01-view ifDescr.10102 included 

snmp-server view agrp-c01-view ifAdminStatus.10102 included

Modify community to include setup view:

snmp-server community agrp-c01-community view agrp-c01-view RW agrp-c01-acl

Verify:

sh run | incl  snmp

Test with (from any of the cluster nodes):

snmpwalk -v 2c -c agrp-c01-community 10.10.53.12 # only configured OIDs will be shown
Also test fence_ifmib itself:
fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel3 --action list # only names of the needed port must be shown

Setup Pacemaker Stonith SNMP

Here I'll be using LACP Port-channel if you don't use LACP, just create one fence_ifmib fence device per port, surely with different names. i.e. fence_ifmib_n01_1-0-1 for GigabitEthernet1/0/1, fence_ifmib_n02_1-0-3 for GigabitEthernet1/0/3 etc.

Create fence device fence_ifmib_n01 for agrp-c01n01:

pcs stonith create fence_ifmib_n01 fence_ifmib pcmk_host_list="agrp-c01n01" ipaddr="agrp-stack01" snmp_version="2c" community="agrp-c01-community" inet4_only="1" port="Port-channel2"  power_wait=4 delay=15 op monitor interval=60s

Create fence device fence_ifmib_n02 for agrp-c01n02:

pcs stonith create fence_ifmib_n02 fence_ifmib pcmk_host_list="agrp-c01n02" ipaddr="agrp-stack01" snmp_version="2c" community="agrp-c01-community" inet4_only="1" port="Port-channel3"  power_wait=4 op monitor interval=60s

Setup constraints (fence_ifmib_n01 will start on agrp-c01n02 & fence_ifmib_n02 will start on agrp-c01n01):

pcs constraint location add lc_fence_ifmib_n01 fence_ifmib_n01 agrp-c01n01 -INFINITY
pcs constraint location add lc_fence_ifmib_n02 fence_ifmib_n02 agrp-c01n02 -INFINITY

Adding stonith level 2:

pcs stonith level add 2 agrp-c01n01 fence_ifmib_n01
pcs stonith level add 2 agrp-c01n02 fence_ifmib_n02

Review Cluster 13 to understand options.

Verify that now GFS2 and CLVMD works even if one node loses power

To verify that:

1. pcs cluster start --all
2. then pull power cable out of one of the nodes
3. dlm_tool ls # you'll see that:
1. new change    member 1 joined 0 remove 1 failed 1 seq 2,2
2. new status    wait fencing

This means that dlm fence was successfull and node is removed (down). You'll see the same messages for both "shared" and "clvmd" lock-spaces. This means that GFS2 file-system and CLVMD will work. 

How to rejoin node to the cluster after above test (I've powered off agrp-c01n02):

fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel3 --action status

fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel3 --action on
ping agrp-c01n02
After successful ping:
pcs cluster start agrp-c01n02

This tutorials were used to understand and setup clustering: 
AN!Cluster
unixarena
redhat.com
Pierky's Blog

Cluster 18. GFS2 (Global File System 2).

GFS2 overview

With DRBD providing the clusters raw storage space, and Clustered LVM providing the logical partitions, we can now look at the clustered file system. This is the role of the GFS2.

It works much like standard filesystem, with user-land tools like mkfs.gfs2, fsck.gfs2 and so on. The major difference is that it and clvmd use the cluster's DLM. Once formatted, the GFS2-formatted partition can be mounted and used by any node in the cluster's closed process group (CPG). All nodes can then safely read from and write to the data on the partition simultaneously.

The Red Hat Global File System (GFS) is Red Hat’s implementation of a concurrent-access shared storage file system. As any such filesystem, GFS allows multiple nodes to access the same storage device, in read/write fashion, simultaneously without risking data corruption. It does so by using a Distributed Lock Manager (DLM) which manages concurrent access from cluster members.

By default, the value of no-quorum-policy is set to stop, indicating that once quorum is lost, all the resources on the remaining partition will immediately be stopped. Typically this default is the safest and most optimal option, but unlike most resources, GFS2 requires quorum to function. When quorum is lost both the applications using the GFS2 mounts and the GFS2 mount itself cannot be correctly stopped. Any attempts to stop these resources without quorum will fail which will ultimately result in the entire cluster being fenced every time quorum is lost.

To address this situation, you can set the no-quorum-policy=freeze when GFS2 is in use. This means that when quorum is lost, the remaining partition will do nothing until quorum is regained:

pcs property set no-quorum-policy=freeze

1. no-quorum-policy=freeze:
1. If quorum is lost, the cluster partition freezes. Resource management is continued: running resources are not stopped (but possibly restarted in response to monitor events), but no further resources are started within the affected partition. This setting is recommended for clusters where certain resources depend on communication with other nodes (for example, OCFS2 mounts). In this case, the default setting no-quorum-policy=stop is not useful, as it would lead to the following scenario: Stopping those resources would not be possible while the peer nodes are unreachable. Instead, an attempt to stop them would eventually time out and cause a stop failure, triggering escalated recovery and fencing.
2. no-quorum-policy=stop (default):
1. If quorum is lost, all resources in the affected cluster partition are stopped in an orderly fashion.

GFS2 setup

From both nodes:

yum install gfs2-utils -y
From one node:

Format the /dev/agrp-c01n01_vg0/shared:

mkfs.gfs2 -j 2 -p lock_dlm -t agrp-c01:shared /dev/agrp-c01n01_vg0/shared # "say" yes to all questions (cluster must be started at the moment of the LV formatting)

The following switches are used with our mkfs.gfs2 call:

• -j 2 # This tells GFS2 to create two journals. This must match the number of nodes that will try to mount this partition at any one time.
• -p lock_dlm # This tells GFS2 to use DLM for its clustered locking.
• -t agrp-c01:shared # This is the lock space name, which must be in the format <cluster_name>:<file-system_name>. The cluster_name must match the one in pcs config | grep Name. The <file-system_name> has to be unique in the cluster, which is easy for us because we'll only have the one gfs2 file system.

From both nodes:

tunegfs2 -l /dev/agrp-c01n01_vg0/shared # both nodes must see this LV in older versions of gfs,  equivalent to this command is: gfs2_tool sb /dev/an-a05n01_vg0/shared all

From one node:

Configure mount point for GFS2 resource (view pcs resource describe Filesystem output to understand all available options):

pcs resource create sharedfs ocf:heartbeat:Filesystem device="/dev/agrp-c01n01_vg0/shared" directory="/shared" fstype="gfs2" 

pcs resource clone sharedfs clone-max=2 clone-node-max=1 interleave=true ordered=true

From both nodes:

df -h /shared # both nodes must see /shared directory as mounted and 259M must shown as used space. 259M is journals consumed space (number of journals * journal size) on disk

From one node:
pcs constraint order start clvmd-clone then sharedfs-clone
pcs constraint colocation add sharedfs-clone with clvmd-clone

Test /shared from any of the nodes (we'll use agrp-c01n02):

cd /shared

touch test{1..10}

ssh agrp-c01n01 ls -lh /shared

This tutorials were used to understand and setup clustering: 
AN!Cluster
unixarena
clusterlabs.org
redhat.com

Cluster 17. DLM & CLVM (Distributed Lock Manager Clustered Logical Volume Management). 

Clustered LVM

With DRBD providing the raw storage for the cluster, we must next consider partitions. This is where Clustered LVM, known as CLVM, comes into play.
CLVM is ideal in that by using DLM, the distributed lock manager. It won't allow access to cluster members outside of corosync's closed process group, which, in turn, requires quorum.
It is ideal because it can take one or more raw devices, known as "physical volumes", or simple as PVs, and combine their raw space into one or more "volume groups", known as VGs. These volume groups then act just like a typical hard drive and can be "partitioned" into one or more "logical volumes", known as LVs. These LVs are where KVM's virtual machine guests will exist and where we will create our GFS2 clustered file system (KVM and GFS2 will be set up in further posts).
LVM is particularly attractive because of how flexible it is. We can easily add new physical volumes later, and then grow an existing volume group to use the new space. This new space can then be given to existing logical volumes, or entirely new logical volumes can be created. This can all be done while the cluster is online offering an upgrade path with no down time.

Installation and initial setup

On both nodes (you can use ssh agrp-c01n01 comand to execute the same commands on the remote node):

yum install dlm lvm2-cluster -y
rsync -av /etc/lvm /root/backups/

Before creation of the clustered LVM, we need to first make some changes to the LVM configuration (vi /etc/lvm/lvm.conf):

1. We need to filter out the DRBD backing devices so that LVM doesn't see the same signature a second time on the DRBD resource's backing device. Or in other words - limit the block devices that are used by LVM commands:
1. filter = [ "a|/dev/drbd|", "a|/dev/sdb|", "r|.*|" ]
2. pvs # should only show drbd and sdb devices
2. Switch from local locking to clustered locking:
1. lvmconf --enable-cluster # Set locking_type to the default clustered type on this system
2. verify: cat /etc/lvm/lvm.conf |grep locking_type |grep -v "#" # must be locking_type = 3 (clustered locking using DLM)
3. Other than this setup, creating LVM logical volumes in a clustered environment is identical to creating LVM logical volumes on a single node. There is no difference in the LVM commands themselves, or in the LVM GUI interface.
3. Do this setting only if your OS itself doesn't use LVM. Don't use locking_type 1 (local) if locking_type 2 or 3 fail. (If an attempt to initialise type 2 or type 3 locking failed, perhaps because cluster components such as clvmd are not running, with this enabled (set to 1), an attempt will be made to use local file-based locking (type 1). If this succeeds, only commands against local VGs will proceed. VGs marked as clustered will be ignored.
1. fallback_to_local_locking = 0
2. verify: cat /etc/lvm/lvm.conf |grep fallback_to_local_locking |grep -v "#"
4. Disable the writing of LVM cache and remove any existing cache:
1. write_cache_state = 0 # default is "1"
2. rm /etc/lvm/cache/*
5. With releases of lvm2 that provide support for lvm2-lvmetad, clusters sharing access to LVM volumes must have lvm2-lvmetad disabled in the configuration and as a service to prevent problems resulting from inconsistent metadata caching throughout the cluster:
1. use_lvmetad = 0
2. verify: cat /etc/lvm/lvm.conf |grep use_lvmetad |grep -v "#"
3. systemctl disable lvm2-lvmetad.service
4. systemctl disable lvm2-lvmetad.socket
5. systemctl stop lvm2-lvmetad.service
6. systemctl status lvm2-lvmetad
7. Remove lvmetad socket file (if exists): rm '/etc/systemd/system/sockets.target.wants/lvm2-lvmetad.socket'

Setup DLM and CLVM

Create DLM and CLVMD clone cluster Resources (Clone options allows resource to can run on both nodes. In other words clone is Active/Active mode):
pcs resource create dlm ocf:pacemaker:controld op monitor interval=30s on-fail=fence
pcs resource clone dlm clone-max=2 clone-node-max=1 interleave=true ordered=true
pcs resource create clvmd ocf:heartbeat:clvm op monitor interval=30s on-fail=fence
pcs resource clone clvmd clone-max=2 clone-node-max=1 interleave=true ordered=true
Verify:
pcs status

Almost every decision in a Pacemaker cluster, like choosing where a resource should run, is done by comparing scores. Scores are calculated per resource, and the cluster resource manager chooses the node with the highest score for a particular resource. (If a node has a negative score for a resource, the resource cannot run on that node.)

We can manipulate the decisions of the cluster with constraints. Constraints have a score. If a constraint has a score lower than INFINITY, it is only a recommendation. A score of INFINITY means it is a must. Default score is INFINITY. To view current constraints: pcs constraint

Each of our devices must be promoted before starting DLM on that node. Configure the resource order (we want drbd to start and promote to Master first and then we want dlm to start and if successful - clvmd will be started):

pcs constraint order promote ms_drbd_r0 then promote ms_drbd_r1 kind=Mandatory
pcs constraint promote start ms_drbd_r1 then start dlm-clone kind=Mandatory

pcs constraint order start dlm-clone then clvmd-clone kind=Mandatory 

Ordering constraints affect only the ordering of resources; they do not require that the resources
be placed on the same node. If you want resources to be started on the same node and in a
specific order, you need both an ordering constraint and a colocation constraint

A colocation constraint determines that the location of one resource depends on the location of another resource. We need clvmd-clone start on the same node where dlm-clone is stared (this resources are cloned, so both nodes will have dlm and clvmd started):
pcs constraint colocation add ms_drbd_r1 with ms_drbd_r0
pcs constraint colocation add dlm-clone with ms_drbd_r1

pcs constraint colocation add clvmd-clone with dlm-clone

Verify:

pcs constraint

Options described:

• These two settings are the default we using them only for clarity:
• clone-max=2 - how many copies of the resource to start (we'll have 2 copies of both dlm and clvmd)
• clone-node-max=1 - how many copies of the resource can be started on a single node (because of previous setting we'll have 2 copies of each resource and because of this setting only one copy of each resource can be started on one node, so we'll have 4 resources, 2 on each node - dlm & clvmd) 
• on-fail=fence - STONITH the node on which the resource failed.
• interleave=true - If this clone depends on another clone via an ordering constraint, is it allowed to start after the local instance of the other clone starts, rather than wait for all instances of the other clone to start (so node1 will start dlm and then clvmd resource, but if we setup "interleave=false", then node1 will wait until dlm is started on node2 too, and will start clvmd only after that)
• ordered=true - will start copies in serial rather than in parallel
• kind=Mandatory - Always. If first (dlm-clone) does not perform first-action (default first-action is start), then (clvmd-clone) will not be allowed to performed then-action (default then-action equals to the value of the first-action, so default is - start). If first (dlm-clone) is restarted, then (clvmd-clone - if running) will be stopped beforehand and  started afterward.

Check cluster status:

pcs status

Check that DLM is working properly:

dlm_tool ls # name clvmd / members 1 2 / seq 2,2 on one node and 1,1 on the other

Function clvmd_start() calls function clvmd_activate_all() which is basicly "ocf_run vgchange -ay" so if a clustered volume group is used, by default the clvm resource agent will activate it on all nodes.

Setup Clustered LV /shared

On one node: pvscan # scans all supported LVM block devices in the system for PVs - we must see only OS PVs's
On one node: pvcreate /dev/drbd{0,1}

On both nodes: pvdisplay # verify this on both nodes smth. like ""/dev/drbd1" is a new physical volume of "<465.58 GiB"" will appear
On both nodes: vgscan # scans all supported LVM block devices in the system for VGs - we must see only OS VGs's

Resource r0 will provide disk space for VMs that will normally run on agrp-c01n01.
Resource r1 will provide disk space for VMs that will normally run on agrp-c01n02.
So we'll use appropriate names while creating VGs:

1. r0 (drbd0) will be in VG agrp-c01n01_vg0
2. r1 (drbd1) will be in VG agrp-c01n02_vg0

On one node: vgcreate -Ay -cy agrp-c01n01_vg0 /dev/drbd0 

On one node: vgcreate -Ay -cy agrp-c01n02_vg0 /dev/drbd1

Options:

• -A - Specifies if metadata should be backed up automatically after a change.  Enabling this is strongly advised! See vgcfgbackup(8) for more information.
• -c - Create a clustered VG using clvmd if LVM is compiled with cluster support.  This allows multiple hosts to share a VG on shared devices.  clvmd and a lock manager must be configured and running.  (A clustered VG using clvmd is different from a shared VG using lvmlockd.)  See clvmd(8) for more information about clustered VGs.

On both nodes: vgs and then pvs
On both nodes: lvscan #  List all logical volumes in all volume groups - we must see only OS LVs's
On one node: lvcreate -L 20G -n shared agrp-c01n01_vg0 # we'll use this shared LV to store OS images and other service information
On both nodes: lvdisplay # we must see newly created "shared" named LV

Now we done with DLM & CLVM. 

Reboot both servers, then start cluster and verify that PV, VG and LV are show properly. Then stop cluster on any node and verify that this node sees only local LVs. If everything works as expected, proceed next.

This tutorials were used to understand and setup clustering: 
AN!Cluster
clusterlabs.org
redhat.com