Cluster 13. Setup IPMI fencing

STONITH (Shoot The Other Node In The Head aka. fencing) protects your data from being corrupted

by non-clustered (because of some problem) nodes or unintended concurrent access.

Just because a node is unresponsive doesn’t mean it has stopped accessing your data. The only

way to be 100% sure that your data is safe, is to use STONITH to ensure that the node is truly offline

before allowing the data to be accessed from another node.

STONITH also has a role to play in the event that a clustered service cannot be stopped. In this case,

the cluster uses STONITH to force the whole node offline, thereby making it safe to start the service

elsewhere.

In pacemaker STONITH is one of the primitive resources.

Testing fence agent

On each node find proper hostname to use with cluster:

crm_node -n # displays the name used by a running cluster

For now we'll be using IPMI fencing - simply powering off the lost server.

Verify that we have needed fencing agent:
[root@agrp-c01n01 ~]# pcs stonith list fence_ipmilan
fence_ipmilan - Fence agent for IPMI

Verify that agent is working properly (from both nodes we test each other - detailed explanation of fence_ipmilan arguments can be found via - man fence_ipmilan):
[root@agrp-c01n01 ~]# fence_ipmilan --ip=agrp-c01n02.ipmi --username="Administrator" --password="our_pass_here" --action="status" --lanplus
[root@agrp-c01n02 ~]# fence_ipmilan --ip=agrp-c01n01.ipmi --username="Administrator" --password="our_pass_here" --action="status" --lanplus

If both tests are successful, proceed.

Setup fencing devices

[root@agrp-c01n01 ~]# pcs stonith describe fence_ipmilan # shows all available options

From one of the nodes (view pcs stonith create help):

Create fence device ipmi_n01 for agrp-c01n01:

pcs stonith create fence_ipmi_n01 fence_ipmilan pcmk_host_list="agrp-c01n01" ipaddr="agrp-c01n01.ipmi" login=Administrator passwd=our_pass_here lanplus=true power_wait=4  delay=15 op monitor interval=60s

Create fence device ipmi_n02 for agrp-c01n02:

pcs stonith create fence_ipmi_n02 fence_ipmilan pcmk_host_list="agrp-c01n02" ipaddr="agrp-c01n02.ipmi" login=Administrator passwd=our_pass_here lanplus=true power_wait=4 op monitor interval=60s

Described (view pcs stonith create help):

• pcs stonith create fence_ipmi_n01 fence_ipmilan - create stonith device named "fence_ipmi_n01" and using "fence_ipmilan" fencing agent
• pcmk_host_list - a list of machines controlled by this device
• ipaddr - proper IPMI-hostname or IPMI IP address of the node
• login & passwd - IPMI username and password
• lanplus - enable enhanced security while accessing IPMI
• power_wait - wait X seconds after issuing ON/OFF
• delay - this only needed on one of the nodes because if both nodes are alive then each will try to fence the other and theoretically both nodes can fence each other at the same time, so giving delay of 15 seconds on one node makes one node be 15 seconds faster than the other in fencing procedure.

Execute pcs config to verify that above commands are inserted properly:
Stonith Devices:
 Resource: fence_ipmi_n01 (class=stonith type=fence_ipmilan)
  Attributes: delay=15 ipaddr=agrp-c01n01.ipmi lanplus=true login=Administrator passwd=our_pass_here pcmk_host_list=agrp-c01n01 power_wait=4
  Operations: monitor interval=60s (fence_ipmi_n01-monitor-interval-60s)
 Resource: fence_ipmi_n02 (class=stonith type=fence_ipmilan)
  Attributes: ipaddr=agrp-c01n02.ipmi lanplus=true login=Administrator passwd=our_pass_here pcmk_host_list=agrp-c01n02 power_wait=4
  Operations: monitor interval=60s (fence_ipmi_n02-monitor-interval-60s)

Verify that both fencing devices are "Started":

pcs stonith 

Enable stonith in a cluster (Run 'man pengine' and 'man crmd' to get a description of the properties):
pcs property set stonith-enabled=true

pcs property # to check all set-up properties

To delete fencing devices (if something went wrong):
pcs stonith delete ipmi_n01
pcs stonith delete ipmi_n02

To add stonith level 1 (stonith levels are run in the order they come: 1st then 2nd up to 9th level - fencing stop trying to fence after first succeeded level). Fencing levels are needed if you have more than one fencing devices/resources in use (view pcs stonith level  add help):
pcs stonith level add 1 agrp-c01n01 fence_ipmi_n01
pcs stonith level add 1 agrp-c01n02 fence_ipmi_n02
pcs stonith level

Fencing resource location:
to verify where fencing resource is started, execute:
pcs stonith 
 fence_ipmi_n01 (stonith:fence_ipmilan): Started agrp-c01n01
 fence_ipmi_n02 (stonith:fence_ipmilan): Started agrp-c01n02
 Target: agrp-c01n01
   Level 1 - fence_ipmi_n01
 Target: agrp-c01n02
   Level 1 - fence_ipmi_n02

As you see  fence_ipmi_n01 is started on agrp-c01n01, and fence_ipmi_n02 is started agrp-c01n02. We want agrp-c01n01 to start fence_ipmi_n02 and agrp-c01n02 to start fence_ipmi_n01. To do so we must first understand what are pacemaker resource constraints. And also we don't want fence_ipmi_n01 be started on agrp-c01n01, and don't want fence_ipmi_n02 be started on agrp-c01n02

Pacemaker resource constraints:

1. Location constraints - tell the cluster which nodes a resource can run on.
2. Ordering Constraints - tell the cluster the order in which resources should start or stop. 
3. Colocation Constraints - tell the cluster that the location of one resource depends on the location of another one.
4. Ticket Constraints - tell the cluster how to coordinate multi-site (geographically-distributed/dispersed clusters) 

To setup stonith preference  with location constraint:
There are two alternative strategies. One way is to say that, by default, resources can run anywhere,
and then the location constraints specify nodes that are not allowed (an opt-out cluster). The other way is to start with nothing able to run anywhere, and use location constraints to selectively enable allowed nodes (an opt-in cluster). We'll use default method:pcs constraint location add lc_ipmi_n02 fence_ipmi_n02 agrp-c01n02 -INFINITY
pcs constraint location add lc_ipmi_n01 fence_ipmi_n01 agrp-c01n01 -INFINITY

Described:

• pcs constraint location add - add new resource location constraint
• lc_ipmi_n01 - location constraint name
• fence_ipmi_n01 - name of the resource 
• agrp-c01n02 -node name for which this location constraint is being setup 
• -INFINITY - score meaning "must never be used to run that resource"

So "pcs constraint location add lc_ipmi_n01 fence_ipmi_n01 agrp-c01n01 -INFINITY" means:

create location constraint named lc_ipmi_n01 which means never run fence_ipmi_n01 resource on agrp-c01n01.

To remove constraint:

pcs constraint --full # or use pcs config
then remove by id:
pcs constraint remove cpecified_id

Execute pcs config to verify that above commands are inserted properly:

Location Constraints:

  Resource: fence_ipmi_n01

    Disabled on: agrp-c01n01 (score:-INFINITY) (id:lc_ipmi_n01)

  Resource: fence_ipmi_n02

    Disabled on: agrp-c01n02 (score:-INFINITY) (id:lc_ipmi_n02)

pcs constraint location 
Location Constraints:
  Resource: fence_ipmi_n01
    Disabled on: agrp-c01n01 (score:-INFINITY)
  Resource: fence_ipmi_n02
    Disabled on: agrp-c01n02 (score:-INFINITY)

Verify that each fencing mechanism is started on the opposite node:
pcs status
 fence_ipmi_n01 (stonith:fence_ipmilan): Started agrp-c01n02
 fence_ipmi_n02 (stonith:fence_ipmilan): Started agrp-c01n01

To view current placement scores:
crm_simulate -sL
Allocation scores:
native_color: fence_ipmi_n01 allocation score on agrp-c01n01: -INFINITY
native_color: fence_ipmi_n01 allocation score on agrp-c01n02: 0
native_color: fence_ipmi_n02 allocation score on agrp-c01n01: 0
native_color: fence_ipmi_n02 allocation score on agrp-c01n02: -INFINITY

Test fencing

Make agrp-c01n02 kernel panic:
echo c > /proc/sysrq-trigger #  immediately and completely hangs the kernel. This does not effect the IPMI BMC, so if we've configured everything properly, the surviving node should be able to use fence_ipmilan to reboot the crashed node.

After that agr-c01n01 must fence agrp-c01n02:

grep "Operation reboot of agrp-c01n02 by agrp-c01n01" /var/log/messages # must show "OK" and agrp-c01n02 must reboot

pcs status:

• fisrst shows: 
• Node agrp-c01n02: UNCLEAN (offline)
• fence_ipmi_n01 (stonith:fence_ipmilan): Started agrp-c01n02 (UNCLEAN)
• then (after successful fencing) shows:
• OFFLINE: [ agrp-c01n02 ]
• fence_ipmi_n01 (stonith:fence_ipmilan): Stopped

stonith_admin --history=*  # must show which node fenced the other and when (* means history from all nodes) on the fence initiator node, fence victim will not show this.

We didn't enable corosync and pacemaker services on our nodes and in our cluster, so after the fencing is done we must enable cluster on the fenced node:

pcs cluster start # this command will start corosync and pacemaker on local node, to start cluster on the remote node specify node host-name after start word, to start cluster on all nodes use --all argument, to stop use stop argument

If IPMI fence devices are stopped because of failure, you can:
pcs stonith cleanup ipmi_n01
pcs stonith cleanup ipmi_n02
after clearing all errors.

Make agrp-c01n01 kernel panic:
echo c > /proc/sysrq-trigger 

After that agr-c01n02 must fence agrp-c01n01:

grep "Operation reboot of agrp-c01n01 by agrp-c01n02" /var/log/messages # must show "OK" and agrp-c01n02 must reboot
pcs status:

• fisrst shows: 
• Node agrp-c01n01: UNCLEAN (offline)
• fence_ipmi_n02 (stonith:fence_ipmilan): Started agrp-c01n01 (UNCLEAN)
• then (after successful fencing) shows:
• OFFLINE: [ agrp-c01n01 ]
• fence_ipmi_n02 (stonith:fence_ipmilan): Stopped

stonith_admin --history=*  # must show which node fenced the other and when (* means history from all nodes) on the fence initiator node, fence victim will not show this.

This tutorials were used to understand and setup clustering: 
AN!Cluster