Cluster 16. DRBD Setup.

Installation

On both nodes:

Linbit provides yum repo only with paid support. So we'll use ELRepo (Enterprise Linux Repository) to install DRBD:

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org # imports the public key

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm # installing ELRepo

yum install drbd84-utils.x86_64 kmod-drbd84.x86_64 -y # DRBD90 is also available in ELRepo but, as official docs state:

https://docs.linbit.com/docs/users-guide-9.0/: With current DRBD-9.0 version running in Dual-Primary mode is not recommended (because of lack of testing). In DRBD-9.1 it will be possible to have more than two primaries at the same time.

systemctl disable drbd.service

systemctl status drbd.service

DRBD will not be able to run under the default SELinux security policies. If you are familiar with

SELinux, you can modify the policies in a more fine-grained manner, but here we will simply exempt

DRBD processes from SELinux control (must be done on both nodes):

semanage permissive -a drbd_t
reboot

Also you can do (must be done on both nodes):

sealert -a /var/log/audit/audit.log
reboot
Then perform suggested actions to solve problems.

Note: This tutorial will create two DRBD resources. Each resource will use a different TCP port. By convention, they start at port 7788 and increment up per resource. So we will be opening ports 7788 and 7789 on each node:

node1:

firewall-cmd --permanent --add-rich-rule='

    rule family="ipv4" 

    source address="10.10.52.2/32" 

    port protocol="tcp" 

    port="7788-7789" accept'

firewall-cmd --reload

firewall-cmd --list-all

node2:
firewall-cmd --permanent --add-rich-rule='

    rule family="ipv4" 

    source address="10.10.52.1/32" 

    port protocol="tcp" 

    port="7788-7789" accept'

firewall-cmd --reload

firewall-cmd --list-all

Setup

Backup existing configs:

rsync -av /etc/drbd.d /root/backups/

cat /etc/drbd.conf 

# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";

include "drbd.d/*.res";

So we need to setup global_common.conf and *.res file for each of our resources.

Setup common DRBD options

vi /etc/drbd.d/global_common.conf # we will describe only changed options

Verify below options:

• in the global section:
• usage-count no; # This tells DRBD that you allow it to report this installation to LINBIT for statistical purposes. If you have privacy concerns, set this to 'no'. 
• in the handlers section:
• fence-peer "/usr/lib/drbd/crm-fence-peer.sh"; # sets constraint (smth like drbd-fence-by-handler-r0-ms_drbd_r0) and there the after-resync-target handler on the peer should remove the constraint again. Thus, if the DRBD replication link becomes disconnected, the crm-fence-peer.sh script contacts the cluster manager, determines the Pacemaker Master/Slave resource associated with this DRBD resource, and ensures that the Master/Slave resource no longer gets promoted on any node other than the currently active one. Conversely, when the connection is re-established and DRBD completes its synchronization process, then that constraint is removed and the cluster manager is free to promote the resource on any node again. In a dual-primary setup, if it was a replication link failure only (if it was node failer - pacemaker will call fence agent on the failed node), and cluster communication is still up, both will call that handler,but only one will succeed to set the constraint. The other will remain IO-blocked, and can optionally "commit suicide" from inside the handler. But just because you where able to shoot the other node does not make your data any better.
• after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh"; # removes constraint after sync
• split-brain "/usr/lib/drbd/notify-split-brain.sh root"; # simply sends message about SB occurrence to the specified e-mail address
• in the startup section:
• become-primary-on both; # This tells DRBD to promote both nodes to Primary on start.
• wait for connection timeouts (wfc) - This command will fail if the device cannot communicate with its partner for timeout seconds. If the peer was working before this node was rebooted, the wfc_timeout is used. If the peer was already down before this node was rebooted, the degr_wfc_timeout is used. If the peer was successfully outdated before this node was rebooted the outdated_wfc_timeout is used. The default value for all those timeout values is 0 which means to wait forever. The unit is seconds. In case the connection status goes down to StandAlone because the peer appeared but the devices had a split brain situation, the default for the command is to terminate:
• wfc-timeout 300;  # This tells DRBD to wait five minutes for the other node to connect. This should be longer than it takes for corosync to timeout and fence the other node *plus* the amount of time it takes the other node to reboot. If you set this too short, you could corrupt your data. If you want to be extra safe, do not use this at all and DRBD will wait for the other node forever. 
• degr-wfc-timeout 120; # This tells DRBD to wait for the other node for three minutes if the other node was degraded the last time it was seen by this node. This is a way to speed up the boot process when the other node is out of commission for an extended duration.
• outdated-wfc-timeout 120; #Same as above, except this time-out is used if the peer was 'Outdated'. 
• in the disk section:
• on-io-error detach; # 
• fencing resource-and-stonith; # This tells DRBD to block IO and fence the remote node (using the 'fence-peer' helper) when connection with the other node is unexpectedly lost. This is what helps prevent split-brain condition and it is incredible important in dual-primary setups! 
• resync-rate 30M; # An eventually running resync process should use about 30MByte/second of IO bandwidth. This tells DRBD how fast to synchronize out-of-sync blocks. The higher this number, the faster an Inconsistent resource will get back to UpToDate state. However, the faster this is, the more of an impact normal application use of the DRBD resource will suffer. We'll set this to 30 MB/sec.
• int the net section:
• protocol C; # tells DRBD not to tell the operating system that the write is complete until the data has reach persistent storage on both nodes. This is the slowest option, but it is also the only one that guarantees consistency between the nodes. It is also required for dual-primary, which we will be using.
• allow-two-primaries; # This tells DRBD to allow two nodes to be Primary at the same time. It is needed when 'become-primary-on both' is set. You only should use this option if you use a shared storage file system on top of DRBD. At the time of writing the only ones are: OCFS2 and GFS. If you use this option with any other file system, you are going to crash your nodes and to corrupt your data! (we are going to use GFS2 and CLVM, both with DLM). This is needed to enable live-migration of our servers. In our case, we'll be running  dual-primary, so we can not safely recover automatically. The only safe option is for the nodes to disconnect from one another and let a human decide which node to invalidate. You can learn more about these options by reading the drbd.conf man page. NOTE! It is not possible to safely recover from a split brain where both nodes were primary. This care requires human intervention, so 'disconnect' is the only safe policy. It doesn't matter what mode you are in, it matters what happened during the time that the nodes were split-brained (time when being StandAlone/UpToDate). If both nodes were secondary during the split-brain, 0pri policy is used. If one node was Primary and the other remained secondary, 1pri policy is used. If both nodes were primary, even for a short time, 2pri is used:
• after-sb-0pri discard-zero-changes; # "after-sb-0pri" - Split brain has just been detected, but at this time the resource is not in the Primary role on any host - neither node is Primary. "discard-zero-changes" - If there is any host on which no changes occurred at all, simply apply all modifications (sync) made on the other and continue. In case none wrote anything this policy uses a random decision to perform a "resync" of 0 blocks. In case both have written something this policy disconnects the nodes.
• after-sb-1pri discard-secondary; # "after-sb-1pri" - Split brain has just been detected, and at this time the resource is in the Primary role on one host. "discard-secondary" - discard changes on the secondary and sync to the primary
• after-sb-2pri disconnect; # "after-sb-2pri" - This tells DRBD what to do in the case of a split-brain when both nodes are primary. "disconnect" - no automatic re-synchronization, simply disconnect.

After setup file on one node (agrp-c01n01), copy file to the other node:
rsync -av /etc/drbd.d/global_common.conf root@agrp-c01n02:/etc/drbd.d/global_common.conf

Setup DRBD resource options

We are going to have 2 resources - r0 and r1, so we must to setup 2 files:

1. resource r0, which will create the device /dev/drbd0, will be backed by each nodes /dev/sda3 partition. It will provide disk space for VMs that will normally run on agrp-c01n01 and provide space for the /shared GFS2 (will be discussed further) partition.
2. resource r1, which will create the device /dev/drbd1, will be backed by each nodes //dev/sda4 partition. It will provide disk space for VMs that will normally run on agrp-c01n02 and provide space for the /shared GFS2 (will be discussed further) partition.

vi /etc/drbd.d/r0.res

# This is the resource used for the shared GFS2 partition and host VMs designed 

# to run on an-a05n01. 

resource r0 {

            # This is the block device path. 

            device /dev/drbd0; 

            # We'll use the normal internal meta-disk. This is where DRBD stores 

            # its state information about the resource. It takes about 32 MB per 

            # 1 TB of raw space. 

            meta-disk internal; 

            # This is the `uname -n` of the first node 

            on agrp-c01n01 { 

                           # The 'address' has to be the IP, not a host name. This is the 

                           # node's SN (sn_bond1) IP. The port number must be unique amoung 

                           # resources. 

                           address 10.10.52.1:7788; 

                           # This is the block device backing this resource on this node. 

                           disk /dev/sda3; 

            } 

           # Now the same information again for the second node. 

          on agrp-c01n02 { 

                           address 10.10.52.2:7788; 

                           disk /dev/sda3; 

          } 

}

Now copy this to r1.res and edit for the agrp-c01n01 VM resource. The main differences are the resource name, r1, the block device, /dev/drbd1, the port, 7789 and the backing block devices, /dev/sda4:

vi /etc/drbd.d/r0.res

# This is the resource used for the shared GFS2 partition and host VMs designed 

# to run on an-a05n01. 

resource r0 {

            # This is the block device path. 

            device /dev/drbd0; 

            # We'll use the normal internal meta-disk. This is where DRBD stores 

            # its state information about the resource. It takes about 32 MB per 

            # 1 TB of raw space. 

            meta-disk internal; 

            # This is the `uname -n` of the first node 

            on agrp-c01n01 { 

                           # The 'address' has to be the IP, not a host name. This is the 

                           # node's SN (sn_bond1) IP. The port number must be unique amoung 

                           # resources. 

                           address 10.10.52.1:7789; 

                           # This is the block device backing this resource on this node. 

                           disk /dev/sda4; 

            } 

           # Now the same information again for the second node. 

          on agrp-c01n02 { 

                           address 10.10.52.2:7789; 

                           disk /dev/sda4; 

          } 

}

Now we will do an initial validation of the configuration - if some options are wrong, descriptive warning message will appear. This is done by running the following command:
drbdadm dump

Now do the same process for node 2 or just use rsync:
rsync -av /etc/drbd.d root@agrp-c01n02:/etc/

After setting node 2 up - verify with drdbadm dump.
To see which options are default options:
drbdsetup /dev/drbd0 show --show-defaults

drbdsetup /dev/drbd1 show --show-defaults

Create DRBD resources

Create DRBD resources (on both nodes):

drbdadm create-md r{0,1} # 'yes' 'yes' => New drbd meta data block successfully created. This step must be completed only on initial device creation. It initializes DRBD’s metadata. If create-md returns "Operation refused" error, then for needed disk (/dev/sda3 or /dev/sda4 in our case):

dd if=/dev/zero of=/dev/sda3 bs=1M count=128

drbdadm up r{0,1} # This step associates the resource with its backing device (or devices, in case of a multi-volume resource), sets replication parameters, and connects the resource to its peer.

DRBD’s virtual status file in the /proc filesystem, /proc/drbd, should now contain information similar to the following (The Inconsistent/Inconsistent disk state is expected at this point):

version: 8.4.10-1 (api:1/proto:86-101)

GIT-hash: a4d5de01fffd7e4cde48a080e2c686f9e8cebf4c build by mockbuild@, 2017-09-15 14:23:22

 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----

    ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:488266148

 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----

    ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:488192424

Description of line with resource number (lines starting with "0:" or "1:" ):

1. cs - connection status (i.e. Connected, Waiting, see http://docs.linbit.com/docs/users-guide-8.4/#s-connection-states)
2. ro - roles (i.e. Primary, Secondary, see http://docs.linbit.com/docs/users-guide-8.4/#s-roles)
3. ds - disk status (i.e. Inconsistent, UpToDate, see http://docs.linbit.com/docs/users-guide-8.4/#s-disk-states)
4. replication protocol mode (A, B or C, see http://docs.linbit.com/docs/users-guide-8.4/#s-replication-protocols)
5. six flags reflecting the I/O status of this resource (i.e. r - running (is the normal state), see http://docs.linbit.com/docs/users-guide-8.4/#s-io-flags) Normally first flag must be "r" and others "-", so flags must be like: r-----
6. the next line after described line is the line of "Performance indicators" (see http://docs.linbit.com/docs/users-guide-8.4/#s-performance-indicators)
7.  oos (out of sync) -  amount of storage currently out of sync; in Kibibytes.

The good thing about DRBD is that we do not have to wait for the resources to be synchronized. So long as one of the resource is UpToDate, both nodes will work. If the Inconsistent node needs to read data, it will simply read it from its peer. But cluster can not be considered redundant until both nodes are UpToDate. So to make disk consistent issue on only on one node (it's the first time we connect our disks, so we really don't need to synchronize disks data):

drbdadm new-current-uuid --clear-bitmap r{0,1} #    You must not use this on pre-existing data!  Even though it may appear to work at first glance, once you switch to the other node, your data is toast, as it never got replicated. So do not leave out the mkfs (or equivalent) - this means that you must create file-system on top of the DRBD device

drbdadm primary --force r{0,1} # just to be sure that synchronization status is checked

Verify (local node is primary because of the drbdadm primary --force r{0,1} command):

cat /proc/drbd

version: 8.4.10-1 (api:1/proto:86-101)

GIT-hash: a4d5de01fffd7e4cde48a080e2c686f9e8cebf4c build by mockbuild@, 2017-09-15 14:23:22

 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:0 nr:0 dw:0 dr:2120 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

    ns:0 nr:0 dw:0 dr:2120 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

As we see both nodes data is UpToDate. Also we can use drbd-overview:

NOTE: drbd-overview will be deprecated soon.

Please consider using drbdtop.

 0:r0/0  Connected Primary/Secondary UpToDate/UpToDate 

 1:r1/0  Connected Primary/Secondary UpToDate/UpToDate 

Or we can use drbdadm status r{0,1}:

r0 role:Primary

  disk:UpToDate

  peer role:Secondary

    replication:Established peer-disk:UpToDate

r1 role:Primary

  disk:UpToDate

  peer role:Secondary

    replication:Established peer-disk:UpToDate

In order for a DRBD resource to be usable, it has to be "promoted". Be default, DRBD resources start in the Secondary state. This means that it will receive changes from the peer, but no changes can be made. You can't even look at the contents of a Secondary resource. Why this is requires more time to discuss than we can go into here.

So the next step is to promote both resource on both nodes.

drbdadm primary r{0,1}

Verify:

cat /proc/drbd # if both nodes are Primary, we're done setting up DRBD.

Pacemaker DRBD resource setup 

Find appropriate type of resource:

pcs resource list | grep -i drbd

ocf:linbit:drbd - Manages a DRBD device as a Master/Slave resource

To view all available non-advanced options:
pcs resource describe drbd

So we need to create Master/Slave resource of type ocf:linbit:drbd:

r0:
pcs resource create drbd_r0 ocf:linbit:drbd drbd_resource=r0 op monitor interval=60s op monitor interval="29s" role="Master" op monitor interval="31s" role="Slave"
pcs resource master ms_drbd_r0 drbd_r0 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true interleave=true
pcs resource cleanup drbd_r0
journalctl | grep -i error

Give it a couple minutes to promote both nodes to Master on both nodes. Initially, it will appear as Master on one node only.

r1:

pcs resource create drbd_r1 ocf:linbit:drbd drbd_resource=r1 op monitor interval=60s op monitor interval="29s" role="Master" op monitor interval="31s" role="Slave"

pcs resource master ms_drbd_r1 drbd_r1 master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

pcs resource cleanup drbd_r1

journalctl | grep -i error

Give it a couple minutes to promote both nodes to Master on both nodes. Initially, it will appear as Master on one node only.

Options description (master-* options are unique to multi-state resource, clone-* options are derived from clone resources):

• clone-max=2 - how many copies of the resource to start
• clone-node-max=1 - how many copies of the resource can be started on a single node
• master-max=2 - how many copies of the resource can be promoted to the master role
• master-node-max=1 - how many copies of the resource can be promoted to the master role on a single node
• notify=true - When stopping or starting a copy of the clone, tell all the other copies beforehand and again when the action was successful. 

We didn't enable drbd daemon at all, so pacemaker will start drbd when cluster is up, and turn it down when cluster stops.

to delete resources:
pcs resource delete drbd_r1 # ms resource will be deleted automatically
pcs resource delete drbd_r0 # ms resource will be deleted automatically
pcs resource cleanup
pcs resource

Split Brain (SB) & Recovery From SB

Normal operation status:
0:r0/0  Connected Primary/Primary UpToDate/UpToDate
or
0:r0/0  Connected Primary/Secondary UpToDate/UpToDate

Master Node WFConnection - primary node can't connect to the secondary node. To resolve see "Manually connecting slave to the master":
0:r0/0  WFConnection Primary/Unknown UpToDate/Unknown

Note: If the master node reports WFConnection while the slave node reports StandAlone, it indicates a DRBD split brain. See "Recovery from SB"

Slave node StandAlone - secondary can't connect to the primary. To resolve see "Manually connecting slave to the master":

0:r0/0  StandAlone Secondary/Unknown UpToDate/Unknown

Note: If the master node reports WFConnection while the slave node reports StandAlone, it indicates a DRBD split brain. See "Recovery from SB"

Both nodes Secondary/Secondary - nodes are connected, but neither is primary. Usually this is due to Pacemaker failer. To resolve restart the entire cluster if DRBD itself is functioning properly (this judgement is up on you):

0:r0/0  Connected Secondary/Secondary UpToDate/UpToDate

0:r0/0  Connected Secondary/Secondary UpToDate/UpToDate

Both nodes StandAlone and Primary  - SB is occured. Both nodes operating independently.  To resolve see "Recovery from SB":

0:r0/0  StandAlone Primary/Unknown UpToDate/Unknown

0:r0/0  StandAlone Primary/Unknown UpToDate/Unknown

"Split-Brain detected, dropping connection!" - message appeared in logs. To resolve see "Recovery from SB".

Manually connecting slave to the master

First identify which node is the master (in such situation Pacemaker and DRBD can see master node differently, we need DRBD point of view).

On the slave node:

drbdadm connect r0 # after that reconnection must take place, verify with drbd-overview

Recovery from SB

When SB occures nodes operating completely independently, because there is no connection and data replication is not taking place. After split brain has been detected, one node will always have the resource in a StandAlone connection state. The other might either also be in the StandAlone state (if both nodes detected the split brain simultaneously), or in WFConnection (if the peer tore down the connection before the other node had a chance to detect split brain).

To understand which node must be considered as master - you need to verify actual data on each node.

First identify which node is considered to be the master and run on the master.

On the slave node:

drbdadm secondary r0

drbdadm connect --discard-my-data connect r0 # Discarding the database on the slave node does not result in a full re-synchronization from master to slave. The slave node has its local modifications rolled back, and modifications made to the master are propagated to the slave. If "Failure: (102) Local address (port) already in use." message is shown issue on the slave node: drbdadm disconnect r0

Verify with drbd-overview - now both nodes must show "Connected", proper role and "UpToDate/UpToDate".

On the master node (SB survivor) - only if it is also StandAlone (if it's WFConnection, this step is not needed):

drbdadm connect r0 # # after that reconnection must take place, verify with drbd-overview

After re-synchronization has completed, the split brain is considered resolved and the two nodes form a fully consistent, redundant replicated storage system again.

This tutorials were used to understand and setup clustering: 
AN!Cluster
LINBIT User's Guide 8.4.x
clusterlabs.org
avid.force.com

Cluster 15. Pacemaker resources & resource constraints.

Pacemaker resources

A resource is a service made highly available by a cluster. The simplest type of resource, a primitive

resource, is described in this section. More complex forms, such as groups and clones, are described

in later sections.

Every primitive resource has a resource agent. A resource agent is an external program that abstracts  the service it provides and present a consistent view to the cluster. Typically, resource agents come in the form of shell scripts. However, they can be written using any technology (such as C, Python or Perl) that the author is comfortable with.

Pacemaker supports several classes of agents (Remember to make sure the computer is not configured to start any services at boot time — that should be controlled by the cluster):

• LSB - LSB resource agents are those found in /etc/init.d (SysV initialization style). Many distributions claim LSB compliance but ship with broken init scripts. Common problematic violations of the LSB standard include:
• Not implementing the status operation at all
• Not observing the correct exit status codes for start/stop/status actions
• Starting a started resource returns an error
• Stopping a stopped resource returns an error
• Systemd - Some newer distributions have replaced the old "SysV" style of initialization daemons and scripts with an alternative called Systemd . Pacemaker is able to manage these services if they are present. Instead of init scripts, systemd has unit files. Generally, the services (unit files) are provided by the OS distribution, but there are online guides for converting from init scripts.
• Upstart - Some newer distributions have replaced the old "SysV" style of initialization daemons (and scripts) with an alternative called Upstart. Instead of init scripts, upstart has jobs. Generally, the services (jobs) are provided by the OS distribution.
• Service - Since there are various types of system services (systemd, upstart, and lsb), Pacemaker supports a special service alias which intelligently figures out which one applies to a given cluster node. This is particularly useful when the cluster contains a mix of systemd, upstart, and lsb. In order, Pacemaker will try to find the named service as:
• an LSB init script
• a Systemd unit file
• an Upstart job
• OCF - the OCF standard is basically an extension of the LSB conventions for init scriptsto support parameters, make them self-describing, and make them extensible. OCF specs have strict definitions of the exit codes that actions must return. The cluster follows these specifications exactly, and giving the wrong exit code will cause the cluster to behave in ways you will likely find puzzling and annoying. In particular, the cluster needs to distinguish a completely stopped resource from one which is in some erroneous and indeterminate state. Parameters are passed to the resource agent as environment variables, with the special prefix OCF_RESKEY_. So, a parameter which the user thinks of as ip will be passed to the resource agent as OCF_RESKEY_ip. The number and purpose of the parameters is left to the resource agent; however, the resource agent should use the meta-data command to advertise any that it supports. The OCF class is the most preferred as it is an industry standard, highly flexible (allowing parameters to be passed to agents in a non-positional manner) and self-describing. pcs resource providers command show which OCF providers presence on the cluster:
• heartbeat
• linbit
• openstack
• pacemaker
• Fencing - this class is used exclusively for fencing-related resources (differently from all other resources which are managed by pcs resource create command, this class is managed by pcs stonith command)
• Nagios Plugins - Nagios Plugins allow us to monitor services on remote hosts. Pacemaker is able to do remote monitoring with the plugins if they are present. A common use case is to configure them as resources belonging to a resource container (usually a virtual machine), and the container will be restarted if any of them has failed. Another use is to configure them as ordinary resources to be used for monitoring hosts or services via the network.

To find which Pacemaker resource classes/standards are supported:

pcs resource standards # List available resource agent standards supported by this installation

lsb

ocf

service

systemd

To view all resources agents:
pcs resource list # this will list all available resource agents with their standard/class names and with provider name (if one is available).

To view usage help for any agent:
pcs resource describe [<standard>:[<provider>:]]<type> [--full]
you can specify only the name of the agent (standard and provider are optional until agent's name in unique). Without "--full" advanced options are not shown.

Pacemaker resource constraints

Pacemaker has below resource constraints:

1. Location constraints - tell the cluster which nodes a resource can run on.
2. Ordering Constraints - tell the cluster the order in which resources should start or stop. 
3. Colocation Constraints - tell the cluster that the location of one resource depends on the location of another one.
4. Ticket Constraints - tell the cluster how to coordinate multi-site (geographically-distributed/dispersed clusters). Apart from local clusters, Pacemaker also supports multi-site clusters. That means you can have multiple, geographically dispersed sites, each with a local cluster. Fail-over between these clusters can be coordinated manually by the administrator, or automatically by a higher-level entity called a Cluster Ticket Registry (CTR). A ticket grants the right to run certain resources on a specific cluster site.

What these all mean:

• If we want to start Apache on node1 we use location constraint, 
• but if we can start Apache on any node but must start it only on the node where MySQL is currently started, then we'll use colocation constraint. 
• if we want Apache to be started only after MySQL is started, then we'll use ordering constraint
• if we have several (not only two) resources that must have applied some rules on them, then we use resources set

Pacemaker non-primitive resources: groups, clones, multi-state, bundles

• groups - One of the most common elements of a cluster is a set of resources that need to be located together, start sequentially, and stop in the reverse order. To simplify this configuration, we support the concept of groups. Resources are started in the order they appear in a group and resources are stopped in the reverse order to which they appear in the group. So group is syntactic-sugar packaging primitive resource, colocation constraints and order constraints under one name (configured by pcs resource group)
• clones - Resources That Get Active on Multiple Hosts (A clone is basically a shortcut: instead of defining n identical, yet differently named resources, a single cloned resource suffices). Three types of cloned resources exist:
• Anonymous - Anonymous clones are the simplest. These behave completely identically everywhere they are running. Because of this, there can be only one copy of an anonymous clone active per machine.
• Globally unique - Globally unique clones are distinct entities. A copy of the clone running on one machine is not equivalent to another instance on another node, nor would any two copies on the same node be equivalent.
• Stateful clones -  multi-state 
• (configured by pcs resource clone)
• multi-state - Resources That Have Multiple Modes - are a specialization of clone resources. Multi-state resources allow the instances to be in one of two operating modes (called roles). The roles are called master and slave, but can mean whatever you wish them to mean. The only limitation is that when an instance is started, it must come up in the slave role (configured by pcs resource master).
• bundles - Pacemaker supports a special syntax for launching a container with any infrastructure it requires: the bundle (configured by pcs resource bundle).

This tutorial was used to understand and setup clustering: 
clusterlabs.org

Cluster 14. Setup partitions for DRBD.

Overview

DRBD states for Distributed Replicated Block Device, is a technology that takes raw storage from two nodes and keeps their data synchronized in real time. It is sometimes described as "network RAID Level 1", and that is conceptually accurate. In this tutorial cluster, DRBD will be used to provide that back-end storage as a cost-effective alternative to a traditional SAN device.

With traditional raid, you would take:

HDD1 + HDD2 -> /dev/sda

With DRBD, you have this:

node1:/dev/sda5 + node2:/dev/sda5 -> both:/dev/drbd0

In both cases, as soon as you create the new device, you pretend like the member devices no longer exist. You format a file-system onto newly created device as an LVM physical volume, and so on.

The main difference with DRBD is that the /dev/drbd0 will always be the same on both nodes. If you write something to node 1, it's instantly available on node 2, and vice versa. Of course, this means that what ever you put on top of DRBD has to be "cluster aware". That is to say, the program or file system using the new /dev/drbd0 device has to understand that the contents of the disk might change because of another node.

Setuping partitions for DRBD

We're going to use a program called parted instead of fdisk. With fdisk, we would have to manually ensure that our partitions fell on 64 KiB block boundaries. With parted, we can use the -a opt to tell it to use optimal alignment, saving us a lot of work. This is important for decent performance performance in our servers. This is true for both traditional platter and modern solid-state drives.
For performance reasons, we want to ensure that the file systems created within a VM matches the block alignment of the underlying storage stack, clear down to the base partitions on /dev/sda (or what ever your lowest-level block device is). By changing the start cylinder of our partitions to always start on 64 KiB boundaries, we're sure to keep the guest OS's file system in-line with the DRBD backing device's blocks. Thus, all reads and writes in the guest OS effect a matching number of real blocks, maximizing disk I/O efficiency.

yum install parted -y # on both nodes

We will setup 2 DRBD resources:

1. r0 - This resource will back the VMs designed to primarily run on agrp-c01n01
2. r1 - This resource will back the VMs designed to primarily run on agrp-c01n02

In the case of the split brain (if both nodes are online and both remain working) we must know which nodes data is more valid. We will consider each node to be more valid for a group of VM's - these VM's will be default resources for that node, so we can easily recover:

1. The DRBD resource hosting agrp-c01n01's servers can invalidate any changes on agrp-c01n02. We consider agrp-c01n01 to be more valid for r0 resource
2. The DRBD resource hosting agrp-c01n02's servers can invalidate any changes on agrp-c01n01. We consider agrp-c01n02 to be more valid for r1 resource

LVM (Logical Volume) - we have physical volumes (execute pvs to view), every PV is then added to the volume group (in the pvs output you can see VG name in front of the PV name), then VG can be divided into the logical volumes (execute vgs to see how many PV and LV each group has and execute lvs to see details about each LV).

We are going to use raw partitions and if your system has not spare raw partitions, in Cluster 1 post I advised to leave space assumed for VMs as raw partition/device, if you didn't the simplest way will be to reinstall all the things from the start, or (steps below are not tested enough, so use them at your own risk - I assume that you have additional partition with enough space to move /root /home and swap there) - at first you will reduce size of /dev/mapper/centos-home partition (yours name can be different - choose non-root partition which is big enough):

Firstly we will resize /home partition:

umount /home

df -h # verify that /home is unmounted

parted -l # to find type of the home LV filesystem (fxs in my case)

lvremove /dev/centos/home # remove LV from the VG

lvcreate -L 50G -n home centos # create new smaller partition for home in VG centos

mkfs.xfs /dev/mapper/centos-home # make home partition formatted as XFS file-system

mount -a # remount all partitions

df -h # verify that /home is mounted

Secondly we will return allocated free space from PV to the raw partition:

Now determine how much space do you want for your DRBD resources, I'll use 500GB for r0 and 500GB for r1:

pvs # find name of the raw partition with enough free space /dev/sda3 in my case

pvdisplay /dev/sda3 # take PE Size (4 MiB in my case) and Allocated PE (27616 in my case). PE is Physical Extent. Calculate space needed for /dev/sda3 (this space is already usedfor /root , /home and swap). You can use bash bc calculator. 4*27616=110464MiBs (MiB = 1024KiB / MB = 1000KB), 110464/1024=107GiBs. Or you can use:

pvs -o +used /dev/sda3 #  this command shows 107.88g instead of our calculated 107GiBs

pvs -v --segments /dev/sda3 # free space must be only at the end of the LVM, in my case I have some free space before root partition, so I need to reallocate space: pvmove --alloc anywhere /dev/sda3:940779-953578 # /dev/sda3:940779-953578 - is the value from PE ranges column of the pvs -v --segments /dev/sda3 command output, after reallocating - recheck with pvs -v --segments /dev/sda3

After reallocating all partitions and having free space only at the end of the LVm disks, move partitions to the additional disk (/dev/sdb2 in my case):

pvmove  /dev/sda3:14816-27615 /dev/sdb2 # moving /root

pvmove  /dev/sda3:2016-14815 /dev/sdb2 # moving /home

pvmove  /dev/sda3:0-2015 /dev/sdb2 # moving swap

If you renamed VG or want to rename - follow instructions in this post - vgrename

vgreduce centos00 /dev/sda3
pvremove /dev/sda3

Lets start with parted:

parted -a optimal /dev/sda # access parted for /dev/sda - /dev/sda3 partition's disk

rm 3 # remove /dev/sda3

print free # to see how many space we have to form new partitions 

mkpart extended 1076MB 501.076GB # 1076MB is the start point of free space 501.076GB is the start-point + 500GB

print free # check that new extended partition is created and it is 500GB

Now we'll create two logical partitions on the newly created extended partition:

mkpart extended 501GB 1001GB # 501GB is the start point of free space 1001GB is the start-point + 500GB
print free # check that new extended partition is created and it is 500GB
We created 2 extended partitions, lets check that they are aligned optimally:
align-check opt 3 # must return "3 aligned"
align-check opt 4 # must return "4 aligned"
quit # to escape parted

This tutorial was used to understand and setup clustering: 
AN!Cluster

ATM / DSL OAM ping

The ATM OAM Ping feature sends an ATM Operation, Administration, and Maintenance (OAM) packet to confirm the connectivity of a specific permanent virtual circuit (PVC). The status of the PVC is displayed when a response to the OAM packet is received. The ATM OAM Ping feature allows the network administrator to verify PVC integrity and facilitates ATM network troubleshooting (it's OAM ping, not IP ping, so that you must be able to success atm ping (OAM ping) without having an IP address set up on an ATM interface (Cisco specific command):

ping atm interface ATM0.1 7 34 # ATM0.1 is ATM subinterface, 7 is VPI, 34 is VCI

Cluster 13. Setup IPMI fencing

STONITH (Shoot The Other Node In The Head aka. fencing) protects your data from being corrupted

by non-clustered (because of some problem) nodes or unintended concurrent access.

Just because a node is unresponsive doesn’t mean it has stopped accessing your data. The only

way to be 100% sure that your data is safe, is to use STONITH to ensure that the node is truly offline

before allowing the data to be accessed from another node.

STONITH also has a role to play in the event that a clustered service cannot be stopped. In this case,

the cluster uses STONITH to force the whole node offline, thereby making it safe to start the service

elsewhere.

In pacemaker STONITH is one of the primitive resources.

Testing fence agent

On each node find proper hostname to use with cluster:

crm_node -n # displays the name used by a running cluster

For now we'll be using IPMI fencing - simply powering off the lost server.

Verify that we have needed fencing agent:
[root@agrp-c01n01 ~]# pcs stonith list fence_ipmilan
fence_ipmilan - Fence agent for IPMI

Verify that agent is working properly (from both nodes we test each other - detailed explanation of fence_ipmilan arguments can be found via - man fence_ipmilan):
[root@agrp-c01n01 ~]# fence_ipmilan --ip=agrp-c01n02.ipmi --username="Administrator" --password="our_pass_here" --action="status" --lanplus
[root@agrp-c01n02 ~]# fence_ipmilan --ip=agrp-c01n01.ipmi --username="Administrator" --password="our_pass_here" --action="status" --lanplus

If both tests are successful, proceed.

Setup fencing devices

[root@agrp-c01n01 ~]# pcs stonith describe fence_ipmilan # shows all available options

From one of the nodes (view pcs stonith create help):

Create fence device ipmi_n01 for agrp-c01n01:

pcs stonith create fence_ipmi_n01 fence_ipmilan pcmk_host_list="agrp-c01n01" ipaddr="agrp-c01n01.ipmi" login=Administrator passwd=our_pass_here lanplus=true power_wait=4  delay=15 op monitor interval=60s

Create fence device ipmi_n02 for agrp-c01n02:

pcs stonith create fence_ipmi_n02 fence_ipmilan pcmk_host_list="agrp-c01n02" ipaddr="agrp-c01n02.ipmi" login=Administrator passwd=our_pass_here lanplus=true power_wait=4 op monitor interval=60s

Described (view pcs stonith create help):

• pcs stonith create fence_ipmi_n01 fence_ipmilan - create stonith device named "fence_ipmi_n01" and using "fence_ipmilan" fencing agent
• pcmk_host_list - a list of machines controlled by this device
• ipaddr - proper IPMI-hostname or IPMI IP address of the node
• login & passwd - IPMI username and password
• lanplus - enable enhanced security while accessing IPMI
• power_wait - wait X seconds after issuing ON/OFF
• delay - this only needed on one of the nodes because if both nodes are alive then each will try to fence the other and theoretically both nodes can fence each other at the same time, so giving delay of 15 seconds on one node makes one node be 15 seconds faster than the other in fencing procedure.

Execute pcs config to verify that above commands are inserted properly:
Stonith Devices:
 Resource: fence_ipmi_n01 (class=stonith type=fence_ipmilan)
  Attributes: delay=15 ipaddr=agrp-c01n01.ipmi lanplus=true login=Administrator passwd=our_pass_here pcmk_host_list=agrp-c01n01 power_wait=4
  Operations: monitor interval=60s (fence_ipmi_n01-monitor-interval-60s)
 Resource: fence_ipmi_n02 (class=stonith type=fence_ipmilan)
  Attributes: ipaddr=agrp-c01n02.ipmi lanplus=true login=Administrator passwd=our_pass_here pcmk_host_list=agrp-c01n02 power_wait=4
  Operations: monitor interval=60s (fence_ipmi_n02-monitor-interval-60s)

Verify that both fencing devices are "Started":

pcs stonith 

Enable stonith in a cluster (Run 'man pengine' and 'man crmd' to get a description of the properties):
pcs property set stonith-enabled=true

pcs property # to check all set-up properties

To delete fencing devices (if something went wrong):
pcs stonith delete ipmi_n01
pcs stonith delete ipmi_n02

To add stonith level 1 (stonith levels are run in the order they come: 1st then 2nd up to 9th level - fencing stop trying to fence after first succeeded level). Fencing levels are needed if you have more than one fencing devices/resources in use (view pcs stonith level  add help):
pcs stonith level add 1 agrp-c01n01 fence_ipmi_n01
pcs stonith level add 1 agrp-c01n02 fence_ipmi_n02
pcs stonith level

Fencing resource location:
to verify where fencing resource is started, execute:
pcs stonith 
 fence_ipmi_n01 (stonith:fence_ipmilan): Started agrp-c01n01
 fence_ipmi_n02 (stonith:fence_ipmilan): Started agrp-c01n02
 Target: agrp-c01n01
   Level 1 - fence_ipmi_n01
 Target: agrp-c01n02
   Level 1 - fence_ipmi_n02

As you see  fence_ipmi_n01 is started on agrp-c01n01, and fence_ipmi_n02 is started agrp-c01n02. We want agrp-c01n01 to start fence_ipmi_n02 and agrp-c01n02 to start fence_ipmi_n01. To do so we must first understand what are pacemaker resource constraints. And also we don't want fence_ipmi_n01 be started on agrp-c01n01, and don't want fence_ipmi_n02 be started on agrp-c01n02

Pacemaker resource constraints:

1. Location constraints - tell the cluster which nodes a resource can run on.
2. Ordering Constraints - tell the cluster the order in which resources should start or stop. 
3. Colocation Constraints - tell the cluster that the location of one resource depends on the location of another one.
4. Ticket Constraints - tell the cluster how to coordinate multi-site (geographically-distributed/dispersed clusters) 

To setup stonith preference  with location constraint:
There are two alternative strategies. One way is to say that, by default, resources can run anywhere,
and then the location constraints specify nodes that are not allowed (an opt-out cluster). The other way is to start with nothing able to run anywhere, and use location constraints to selectively enable allowed nodes (an opt-in cluster). We'll use default method:pcs constraint location add lc_ipmi_n02 fence_ipmi_n02 agrp-c01n02 -INFINITY
pcs constraint location add lc_ipmi_n01 fence_ipmi_n01 agrp-c01n01 -INFINITY

Described:

• pcs constraint location add - add new resource location constraint
• lc_ipmi_n01 - location constraint name
• fence_ipmi_n01 - name of the resource 
• agrp-c01n02 -node name for which this location constraint is being setup 
• -INFINITY - score meaning "must never be used to run that resource"

So "pcs constraint location add lc_ipmi_n01 fence_ipmi_n01 agrp-c01n01 -INFINITY" means:

create location constraint named lc_ipmi_n01 which means never run fence_ipmi_n01 resource on agrp-c01n01.

To remove constraint:

pcs constraint --full # or use pcs config
then remove by id:
pcs constraint remove cpecified_id

Execute pcs config to verify that above commands are inserted properly:

Location Constraints:

  Resource: fence_ipmi_n01

    Disabled on: agrp-c01n01 (score:-INFINITY) (id:lc_ipmi_n01)

  Resource: fence_ipmi_n02

    Disabled on: agrp-c01n02 (score:-INFINITY) (id:lc_ipmi_n02)

pcs constraint location 
Location Constraints:
  Resource: fence_ipmi_n01
    Disabled on: agrp-c01n01 (score:-INFINITY)
  Resource: fence_ipmi_n02
    Disabled on: agrp-c01n02 (score:-INFINITY)

Verify that each fencing mechanism is started on the opposite node:
pcs status
 fence_ipmi_n01 (stonith:fence_ipmilan): Started agrp-c01n02
 fence_ipmi_n02 (stonith:fence_ipmilan): Started agrp-c01n01

To view current placement scores:
crm_simulate -sL
Allocation scores:
native_color: fence_ipmi_n01 allocation score on agrp-c01n01: -INFINITY
native_color: fence_ipmi_n01 allocation score on agrp-c01n02: 0
native_color: fence_ipmi_n02 allocation score on agrp-c01n01: 0
native_color: fence_ipmi_n02 allocation score on agrp-c01n02: -INFINITY

Test fencing

Make agrp-c01n02 kernel panic:
echo c > /proc/sysrq-trigger #  immediately and completely hangs the kernel. This does not effect the IPMI BMC, so if we've configured everything properly, the surviving node should be able to use fence_ipmilan to reboot the crashed node.

After that agr-c01n01 must fence agrp-c01n02:

grep "Operation reboot of agrp-c01n02 by agrp-c01n01" /var/log/messages # must show "OK" and agrp-c01n02 must reboot

pcs status:

• fisrst shows: 
• Node agrp-c01n02: UNCLEAN (offline)
• fence_ipmi_n01 (stonith:fence_ipmilan): Started agrp-c01n02 (UNCLEAN)
• then (after successful fencing) shows:
• OFFLINE: [ agrp-c01n02 ]
• fence_ipmi_n01 (stonith:fence_ipmilan): Stopped

stonith_admin --history=*  # must show which node fenced the other and when (* means history from all nodes) on the fence initiator node, fence victim will not show this.

We didn't enable corosync and pacemaker services on our nodes and in our cluster, so after the fencing is done we must enable cluster on the fenced node:

pcs cluster start # this command will start corosync and pacemaker on local node, to start cluster on the remote node specify node host-name after start word, to start cluster on all nodes use --all argument, to stop use stop argument

If IPMI fence devices are stopped because of failure, you can:
pcs stonith cleanup ipmi_n01
pcs stonith cleanup ipmi_n02
after clearing all errors.

Make agrp-c01n01 kernel panic:
echo c > /proc/sysrq-trigger 

After that agr-c01n02 must fence agrp-c01n01:

grep "Operation reboot of agrp-c01n01 by agrp-c01n02" /var/log/messages # must show "OK" and agrp-c01n02 must reboot
pcs status:

• fisrst shows: 
• Node agrp-c01n01: UNCLEAN (offline)
• fence_ipmi_n02 (stonith:fence_ipmilan): Started agrp-c01n01 (UNCLEAN)
• then (after successful fencing) shows:
• OFFLINE: [ agrp-c01n01 ]
• fence_ipmi_n02 (stonith:fence_ipmilan): Stopped

stonith_admin --history=*  # must show which node fenced the other and when (* means history from all nodes) on the fence initiator node, fence victim will not show this.

This tutorials were used to understand and setup clustering: 
AN!Cluster

Cluster 12. Configuring the Cluster Foundation.

Time sync

Firs we need to keep time in sync between both nodes. Install ntp daemon, set time from pool,ntp,org, start ntp daemon:

yum install ntp -y
ntpdate pool.ntp.org
systemctl -l enable ntpd.service
systemctl -l start ntpd.service

Set proper timezone:
timedatectl list-timezones | grep Baku #you must select proper city
timedatectl set-timezone Asia/Baku
Verify:
timedatectl

Packages

Needed packages:

Corosync - manages cluster communication, quorum and membership, uses totem protocol for "heartbiting". Prior to CentOS 7 crosync itself only cared about who is a cluster member and making sure all members get all totem messages. What happens after the cluster reforms was up to the cluster manager (cman), and the resource group manager (rgmanager). In CentOS 7 cman work (mainly quorum - assign quorum votes and control them) is done by corosync and the rgmanager work is done by pacemaker. (To be frankly - cman work is now done by votequorum part of the corosync).

Pacemaker - cluster resource manager (pcsd is it's daemon)

pcs - (ccs in CentOS 6) - CentOS 7 command line configuration utility

psmisc - contains utilities for managing processes on your system: pstree, killall, and fuser.

policycoreutils-python - contains the core utilities that are required for the basic operation of a Security-Enhanced Linux (SELinux) system and its policies.

fence-agents - provides various agents for fencing (ipmi fence, cisco fence etc.)

dlm - Distributed Lock Manager

Install needed packages:

I prefer to use bash autocomplete: bash-autocomplete-setup-link

yum install -y  corosync pacemaker pcs psmisc policycoreutils-python fence-agents dlm

Setup initial cluster 

Related to pcsd.service:

systemctl -l enable pcsd.service

systemctl -l start pcsd.service

systemctl -l status pcsd.service #pcsd service automatically starts corosync and pacemaker services when needed

Setup password for cluster user (CentOS 6 cluster user was ricci)

passwd hacluster

Setup and verify firewall:

Verify if firewalld is active:

firewall-cmd --state

firewall-cmd --permanent --add-service=high-availability # view /usr/lib/firewalld/services/high-availability.xml to see which port are in high-availability service

firewall-cmd --reload

firewall-cmd --list-all

Ports listed in /usr/lib/firewalld/services/high-availability.xml :

tcp 2224 - PCSD Web UI (High Availability Web Management)
tcp 3121 - Pacemaker Remote
tcp 5403 - needed for corosync-qnetd
udp 5404 - totem protocol multicast
udp 5405 - totem protocol

tcp 21064 - DLM

Login to any of the cluster node and authenticate “hacluster” user. 

We must use names from /etc/hosts which are resolved to our bcn-bond1 addresses (this step automatically setups Corosync authentication):

pcs cluster auth agrp-c01n01 agrp-c01n02 #username is hacluster

To change hacluster password:

passwd hacluster

pcs cluster auth agrp-c01n01 agrp-c01n02 --force  # "--force" will force authentication even node is already authenticated 

To verify:

cat /var/lib/pcsd/tokens # both nodes info must be here

Create new cluster named agrp-c01 (this step automatically setups Corosync cluster membership & also synchronizes configuration):

pcs cluster setup --name agrp-c01 agrp-c01n01 agrp-c01n02 --transport # udpu transport - UDP Unicast

After all executed steps new file is created:

vi /etc/corosync/corosync.conf

secauth: off attribute. This controls whether the cluster communications are encrypted or not. We can safely disable this because we're working on a known-private network, which yields two benefits; It's simpler to setup and it's a lot faster. If you must encrypt the cluster communications, then you can do so here.

pcs cluster start --all # --all option will start cluster on all nodes & also will start corosync.service and pacemaker.service.

If you want corosync and pacemaker to start automatically on boot:

pcs cluster enable --all # this tutorial doesn't use this setting

Check cluster is functioning properly (on both nodes):

systemctl status corosync # must be active/disabled without any error

Use corosync-cfgtool -s to check whether cluster communication is happy (output must have local node's proper IP in id and "no faults" in status):

Printing ring status.

Local node ID 2

RING ID 0

id = 10.10.53.2

status = ring 0 active with no faults

Next, check the membership and quorum APIs (both nodes must join the cluster):

corosync-cmapctl | grep members

runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0

runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.10.53.1)

runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1

runtime.totem.pg.mrp.srp.members.1.status (str) = joined

runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0

runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.10.53.2)

runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1

runtime.totem.pg.mrp.srp.members.2.status (str) = joined

Verify that corosync uses no-multicats (udpu transport - UDP Unicast):

corosync-cmapctl | grep transport

totem.transport (str) = udpu

pcs status corosync

Membership information

----------------------

    Nodeid      Votes Name

         1          1 agrp-c01n01 (local)

         2          1 agrp-c01n02

systemctl status pacemaker # must be active/disabled without any error

Verify that all Pacemaker daemons (pacemaker itself + 6 daemons) are loaded:

ps axf | grep pacemaker 

7588 ?        Ss     0:00 /usr/sbin/pacemakerd -f

 7589 ?        Ss     0:00  \_ /usr/libexec/pacemaker/cib

 7590 ?        Ss     0:00  \_ /usr/libexec/pacemaker/stonithd

 7591 ?        Ss     0:00  \_ /usr/libexec/pacemaker/lrmd

 7592 ?        Ss     0:00  \_ /usr/libexec/pacemaker/attrd

 7593 ?        Ss     0:00  \_ /usr/libexec/pacemaker/pengine

 7594 ?        Ss     0:00  \_ /usr/libexec/pacemaker/crmd

Finally check cluster overall status:

pcs status # both nodes must be online (it can take several minutes to become online)

Finally, ensure there are no startup errors (aside from messages relating to not having STONITH

configured, which are OK at this point):

journalctl | grep -i error

DC - this string is seen in the "pcs status" output

Designated Coordinator (DC) One CRM in the cluster is elected as Designated Coordinator (DC). The DC is the only entity in the cluster that can decide that a cluster wide change needs to be performed, such as fencing a node or moving resources around. The DC is also the node where the master copy of the CIB is kept. All other nodes get their configuration and resource allocation information from the current DC. The DC is elected from all nodes in the cluster after a membership change (lost nodes etc.).

Quorum:

The votequorum service is part of the corosync project. This service can be optionally loaded into the nodes of a corosync cluster to avoid split-brain situations. It does this by having a number of votes assigned to each system in the cluster and ensuring that only when a majority of the votes are present, cluster operations are allowed to proceed. The service must be loaded into all nodes or none. If it is loaded into a subset of cluster nodes the results will be unpredictable. The following corosync.conf extract will enable votequorum service within corosync: 

quorum { provider: corosync_votequorum } # verify with pcs cluster corosync | grep provider

votequorum reads its configuration from corosync.conf. Some values can be changed at runtime, others are only read at corosync startup. It is very important that those values are consistent across all the nodes participating in the cluster or votequorum behavior will be unpredictable.

The "two node cluster" is a use case that requires special consideration. With a standard two node cluster, each node with a single vote, there are 2 votes in the cluster. Using the simple majority calculation (50% of the votes + 1) to calculate quorum, the quorum would be 2. This means that the both nodes would always have to be alive for the cluster to be quorate and operate. Enabling two_node: 1, quorum is set artificially to 1. So simply saying, with two_node=1 one node will continue when the other node fails. The way it works is that in the event of a network outage both nodes race in an attempt to fence each other and the first to succeed continues in the cluster. The system administrator can also associate a delay with a fencing agent so that one node can be given priority in this situation so that it always wins the race. Also this delay will help to escape fence-looping.
pcs cluster corosync  | grep two_node
and
pcs quorum | grep "Flags\|Quorum" # it's like "Flags: 2Node" and "Quorum: 1"

two_node=1 requires expected_votes is set to 2 (pcs quorum status | grep Expected votes) and it's automatically set to this value (2) automatically when two node cluster is set up. Also this setting (two_node=1) considering that you have proper fencing setup - 

wait_for_all (pcs quorum | grep flags # it's like WaitForAll) - When enabled, the cluster will be quorate for the FIRST TIME only after all nodes have been visible at least once at the same time. The wait_for_all option is automatically enabled when a cluster has two nodes, does not use a quorum device, and auto_tie_breaker is disabled. You can override this by explicitly setting wait_for_all to 0 but in two-node cluster this is not recommended.
auto_tie_breaker When enabled, the cluster can suffer up to 50% of the nodes failing at the same time, in a deterministic fashion. auto_tie_breaker is not compatible with two_node as both are systems for determining what happens should there be an even split of nodes. If you have both enabled, then an error message will be issued and two_node will be disabled.
You can verify wait_for_all setting effect:
pcs cluster stop --all
pcs cluster start
pcs quorum status | grep "Qurum:" # you'll get "Quorum: Activity blocked"
Now:
pcs cluster start --all
pcs quorum status | grep "Qurum:" # you'll get "Quorum: 1"
pcs cluster stop here_name_of_the_other_node
pcs quorum status | grep "Qurum:" # you'll get "Quorum: 1"
So as you see - as expected, wait_for_all only needs all nodes to be online when starting the first time.

How to start nodes when you know the cluster is inquorate, but you are confident that the cluster should proceed with resource management regardless. It can be when one node is powered off and the other node didn't start cluster before first node power off. But you must be sure that the other node doesn't have access to the resources:
pcs quorum unblock # this disables wait_for_all option and then re-enables it

Location

Overall cluster configuration in xml format can be retrieved:

pcs cluster cib scope=configuration # cluster XML dump file, where scope is one from the list: configuration, crm_config, nodes, resources, constraints, status

Note: By default symmetric-cluster is created, meaning  all resources can run anywhere:

There are two alternative strategies. One way is to say that, by default, resources can run anywhere, and then the location constraints specify nodes that are not allowed (an opt-out cluster). The other way is to start with nothing able to run anywhere, and use location constraints to selectively enable allowed nodes (an opt-in cluster).

Destroying cluster

If something went wrong, you can destroy cluster and all it's configuration:
pcs cluster destroy --all
rm -rf /var/lib/pacemaker
rm -rf /var/lib/pcsd
rm -rf /etc/corosync

This tutorials were used to understand and setup clustering: 
AN!Cluster
unixarena
people.redhat.com