Tuesday, July 3, 2018

ASA 1. Active/Standby Failover.

1. Small FAQ

ASA Services Module is not considered in this blog post.
Failover can be Active/Active or Active/Standby. Active/Active failover must be setup in multi-context mode (per security context) and doesn't support VPN failover. Active/Standby failover supports VPN failover but all traffic goes only through ASA in active role (load-balancing is not supported). Primary and secondary units doesn't change their types (primary or secondary), only their state/role can change (i.e. secondary unit can be in active state/role due to primary unit fail).
Units have one dedicated physical port to be used as failover control link, this links must be interconnected (back-to-back without an intermediate switch). Failover control link is used for:
  1. initial failover peer discovery and negotiation
  2. replication of the configuration from active to the standby peer
  3. unit health monitoring
Both Active/Active and Active/Standby failover can be configured in stateless (no connections states are tracked) or stateful (packets and connections states are tracked and connections are not dropped when failover is done) manner. By default failover operates in stateless manner. To support stateful failover - Stateful Link must be setup.
Active unit accepts configuration changes and places the same commands to the standby unit, no configuration changes must be performed on a standby unit. If stateful failover is configured - active ASA monitors, builds and tears down all connections. Also this info also tracked and synchronized:
  1. stateful table for UDP and TCP connections
  2. ARP table and MAC mapping table
  3. routing table
  4. certain application inspection data
  5. most VPN data structures (only some client-less VPN info remains stateless)
When Active/Standby failover is used - for a  switchover to occur automatically - the active unit must become less operational than standby unit, at least one of following must occur:

  1. one of the internal (monitored) interfaces goes down
  2. an interface expansions slot fails
  3. an IPS, CSC or CX application module fails

Health messages by default are exchanged in 1 second interval. If failover control link fails, failover becomes disabled. By default, a switchover occurs when at least one interface on the active unit or within an active failover group fails.

Failover provides very effective first-hop redundancy capabilities by allowing the MAC and IP address pair on each data interface to move between the failover peers based on which unit is active at any given time. Because all physical interface connections and their configurations are identical between the members of a failover pair, active ASA unit switchovers are completely transparent to the adjacent network devices and endpoints. When you enable failover, the IP address configured on each data interface becomes the active one. When the active unit fails, the standby peer automatically assumes ownership of these addresses upon taking over the active role and seamlessly picks up transit traffic processing.

In Active/Standby failover secondary unit in active state remains active even if primary unit becomes operationally healthy. The primary unit takes over an active role only if secondary unit becomes unhealthy or if switchover is done manually.

All configuration changes must be done on the active unit (either in Primary or Secondary state).

2. Preparation

  1. When grouping two devices in failover, the following hardware parameters must be identical:
    1. exact model number
    2. number and type of physical interfaces must be the same (also expansion modules must be the same if any)
    3. all cables must be connected appropriate to the Layer 2 on both units for unit health monitoring to be held properly
    4. all hardware or software modules and software must be the same on both units
    5. amount of RAM and system flash must be the same on both units
    6. both failover peers should run the same software image during normal operation (different images are supported during upgrade) 
    7. prior to ASA8.3(1) licence features on both units must to be the same
    8. Cisco ASA 5505, ASA 5510, and ASA 5512-X appliances must have the Security Plus license installed.
    9. The state of the Encryption-3DES-AES license must match between the units. In other words, it must be either disabled or enabled on both failover peers.
  2. Choose roles for each ASA- one ASA will be primary and the other - secondary (i.e. old ASA - primary / new ASA - secondary). 
  3. Dedicate one physical interface (the same, i.e. Gi0/3 on both) on each unit for the failover control link and connect them back-to-back without an intermediate switch.  
  4. If you plan to use stateful failover - dedicate another physical interface to be used as stateful link
  5. Choose IP addresses for the primary and secondary units, used failover subnet cannot overlap with any data interfaces (one subnet per failover control and failover state links)
  6. Choose security key to encrypt failover traffic

3. Setup

3.1 Setup with separate physical interface for Failover Link and State Link

Start failover configuration on the primary node (also consider maintenance window as interface will go down while transiting to the failover active state - it takes roughly 1 minute to go into active state):
interface GigabitEthernet0/2
 no shutdown
interface GigabitEthernet0/3
 no shutdown
failover lan unit primary 
failover lan interface FailoverControl GigabitEthernet0/2 
failover link FailoverState GigabitEthernet0/3 
failover interface ip FailoverControl 172.20.0.1 255.255.255.0 standby 172.20.0.2 
failover interface ip FailoverState 172.20.1.1 255.255.255.0 standby 172.20.1.2 
failover ipsec pre-shared-key *****
failover 
     No Active mate detected
show failover | grep host
     This host: Primary - Active
     Other host: Secondary - Not Detected

Then configure failover on standby unit:
interface GigabitEthernet0/2
 no shutdown
interface GigabitEthernet0/3
 no shutdown
failover lan unit secondary 
failover lan interface FailoverControl GigabitEthernet0/2 
failover replication http 
failover link FailoverState GigabitEthernet0/3 
failover interface ip FailoverControl 172.20.0.1 255.255.255.0 standby 172.20.0.2 
failover interface ip FailoverState 172.20.1.1 255.255.255.0 standby 172.20.1.2 
failover ipsec pre-shared-key *****
failover 
     Detected an Active mate
     Beginning configuration replication from mate. 
     End configuration replication from mate.
show failover | grep host
     This host: Secondary - Standby Ready
     Other host: Other host: Primary - Active

The failover key command enables password failover encryption. Use either a string of letters, numbers, and punctuation with 1 to 63 characters or a hexadecimal value of up to 32 digits. Only use this option when running Cisco ASA Software versions earlier than 9.1(2) or deploying stateless failover.
IPSec site-to-site tunnel is more secure approach to failover link protection, so always use it in Cisco ASA Software version 9.1(2) and later. The failover ipsec pre-shared-key command enables this method of failover encryption. You must deploy stateful failover to use this feature. When using IPSec as encryption method - this tunnel is not counted in ASA maximum supported VPN count.

3.2 Setup with 1 redundant interface for both Failover Link and State Link

If you want to use redundant interface:
On primary unit:
interface GigabitEthernet0/2
 no shutdown
interface GigabitEthernet0/3
 no shutdown
interface Redundant 1
  member-interface GigabitEthernet 0/2
  INFO: security-level and IP address are cleared on GigabitEthernet0/2
  member-interface GigabitEthernet 0/3
  INFO: security-level and IP address are cleared on GigabitEthernet0/3
failover lan unit primary 
failover lan interface FailoverLink Redundant1
INFO: Non-failover interface config is cleared on Redundant1 and its sub-interfaces
failover interface ip FailoverLink 172.20.0.1 255.255.255.0 standby 172.20.0.2
failover link FailoverLink
failover ipsec pre-shared-key *****
failover 
     No Active mate detected
show failover | grep host
     This host: Primary - Active
     Other host: Secondary - Not Detected

Then configure failover on standby unit:
interface GigabitEthernet0/2
 no shutdown
interface GigabitEthernet0/3
 no shutdown
interface Redundant 1
  member-interface GigabitEthernet 0/2
  INFO: security-level and IP address are cleared on GigabitEthernet0/2
  member-interface GigabitEthernet 0/3
  INFO: security-level and IP address are cleared on GigabitEthernet0/3
failover lan unit secondary 
failover lan interface FailoverLink Redundant1
INFO: Non-failover interface config is cleared on Redundant1 and its sub-interfaces
failover interface ip FailoverLink 172.20.0.1 255.255.255.0 standby 172.20.0.2
failover link FailoverLink
failover ipsec pre-shared-key *****
failover 
     Detected an Active mate
     Beginning configuration replication from mate.
     End configuration replication from mate.
show failover | grep host
     This host: Secondary - Standby Ready
     Other host: Other host: Primary - Active

3.3 Disabling failover monitoring for interface

You can have an interface which can't be replicated (i.e. like fiber optic coming directly from an ISP). This interfaces must be unplugged from failed ASA and then plugged into currently active ASA. To exclude interface from the failover monitoring:
asa (config)# no monitor interface interface_name_here
By default, monitoring physical interfaces is enabled and monitoring subinterfaces is disabled. You can check this via: sh run all | grep monitor-interface

If you can replicate your interface (ex. Gi0/0), then on primary node:
# conf t
# int gi0/0
# ip address 192.168.0.1 255.255.255.0 standby 192.168.0.2
To check:
find name of the gi0/0:
sh nameif | grep G.*0/0
GigabitEthernet0/0       inside                   100
Check this name in failover:
sh failover | grep inside
  Interface inside (192.168.0.1): Normal (Monitored)
  Interface inside (192.168.0.2): Normal (Monitored)

You also can use standby IP to access node in Standby state.

4. Test and operate

Use the show failover command to monitor the operational state of the failover.
You can use show failover history command to investigate failover events.

Use failover execute mate command_to_execute_remotely to execute command on the standby unit (i.e.: failover exec mate show version | grep Serial). Do not execute configuration commands on the standby unit.

Use write standby - to restore standby unit proper state after accidentally performing configuration on a standby unit, this command replaces all configuration with the copy of the configuration from the active unit.

Use the failover active command on the standby unit to transit unit to the active state.
Use the no failover active command on the currently active unit  to transit unit to the standby state.

Thursday, May 17, 2018

Cluster 24. Resizing LVM image of the Virtual Machine.

On guest:
fdisk -l /dev/vda

Resize LV to be 20GB:
lvresize -L 21474836480b /dev/agrp-c01n02_vg0/vm-test
  Size of logical volume agrp-c01n02_vg0/vm-test changed from 15.00 GiB (3840 extents) to 20.00 GiB (5120 extents).
  Logical volume agrp-c01n02_vg0/vm-test successfully resized.

Find domain ID:
virsh list
 Id    Name                           State
----------------------------------------------------
 1     vm-test                 running

Resize vda of the guest:
virsh blockresize  --path /dev/agrp-c01n02_vg0/vm-test --size 21474836480b  1
Block device '/dev/agrp-c01n02_vg0/vm-test' is resized

On guest - check that vda is resized to the needed size:
lsblk -a | grep ^vda 
vda                         252:0    0   20G  0 disk 

View last partition index (here it's 2 because of vda2) on the guest:
lsblk -a | grep vda
vda                         252:0    0   20G  0 disk 
├─vda1                      252:1    0  500M  0 part /boot
└─vda2                      252:2    0 14.5G  0 part 

Find end of the last partition (text in bold is end - 31208) on the guest:
fdisk -l

Disk /dev/vda: 21.5 GB, 21474836480 bytes
16 heads, 63 sectors/track, 41610 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a4595

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           3        1018      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2            1018       31208    15215616   8e  Linux LVM


Create new partition vda3 on the guest:
fdisk /dev/vda 
input n letter # for creating new partition
input p # for primary partition 
input 3 # for new partition index
input 31208 # for "First cylinder (1-41610, default 1):"
enter # to accept "Last cylinder, +cylinders or +size{K,M,G} (31208-41610, default 41610):"
input w # to save 

View current partition table on the guest:
partx -l /dev/vda
Re-read partition table:
partx -a /dev/vda # this will give "BLKPG: Device or resource busy" for already added partition
Check that we have device nodes for /dev/sdb itself and the partitions on it:
ls /dev/vda*
/dev/vda  /dev/vda1  /dev/vda2 /dev/vda3

Create PV and verify on the guest:
pvcreate /dev/vda3
  Physical volume "/dev/vda3" successfully created
pvs
 PV         VG       Fmt  Attr PSize  PFree
  /dev/vda2  VolGroup lvm2 a--  14.51g    0 
  /dev/vda3           lvm2 ---   5.00g 5.00g
Extend VG with new PV nad verify:
vgextend VolGroup /dev/vda3
  Volume group "VolGroup" successfully extended
pvs
  PV         VG       Fmt  Attr PSize  PFree
  /dev/vda2  VolGroup lvm2 a--  14.51g    0 
  /dev/vda3  VolGroup lvm2 a--   5.00g 5.00g

Extend LV size to the all available free space on the guest:
lvextend -l +100%FREE /dev/VolGroup/lv_root 
  Size of logical volume VolGroup/lv_root changed from 13.01 GiB (3330 extents) to 18.00 GiB (4609 extents).
  Logical volume lv_root successfully resized

Verify on the guest:
lvs
  LV      VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_root VolGroup -wi-ao---- 18.00g                                                    
  lv_swap VolGroup -wi-ao----  1.50g 

Resize file system on the guest:
For ext type filesystem:
resize2fs /dev/VolGroup/lv_root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/VolGroup/lv_root is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 2
Performing an on-line resize of /dev/VolGroup/lv_root to 4719616 (4k) blocks.
The filesystem on /dev/VolGroup/lv_root is now 4719616 blocks long.
For xfs filesystem:
xfs_growfs /dev/VolGroup/lv_root


Verify on guest:
df -h
Filesystem                                   Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root      18G  9.3G  7.5G  56% /
tmpfs                                           939M     0  939M   0% /dev/shm
/dev/vda1                                    477M   48M  405M  11% /boot

Thursday, May 10, 2018

Cluster 23. Converting qcow2 to LVM storage.

If you want target machine to remain working while we changing it's storage (it's only recommended for VM's which are NOT having their data updated continuously - like NTP server, if your VM data is updated continuously - you MUST stop the VM and then convert VM's storage and then start VM on the new storage, otherwise you will lose data):

virsh # entry to the virsh console
list # list all running VMs
 Id    Name                           State
----------------------------------------------------
 4     rntp                           running
 5     rftp                           running
 18    agisrv                         running
 19    squid                          running
 23    cc_replica                     running
 27    nagios                         running
 29    rvoip                          running
domblklist rntp # list block-devices for given VM name
Target     Source
------------------------------------------------
vda        /var/lib/libvirt/images/rntp.qcow2
hdc        -
domblkinfo rntp vda # view size for an image (we need "Capacity")
Capacity:       16106127360
Allocation:     1576611840
Physical:       1576611840
or:
qemu-img info /var/lib/libvirt/images/rntp.qcow2 # we need "virtual size"
image: /var/lib/libvirt/images/rntp.qcow2
file format: qcow2
virtual size: 15G (16106127360 bytes)
disk size: 1.5G
cluster_size: 65536

Create LV with needed size:
lvcreate -L 16106127360b -n vm01-rntp_0 agrp-c01n01_vg0

virsh dumpxml rntp > /root/rntp.xml
Calculate md5 checksum (md5 sums only verifies/works with the file content rather than the file name):
md5sum /var/lib/libvirt/images/rntp.qcow2 > rntp.md5
md5sum /root/rntp.xml >> rntp.md5

Copy qcow2 image to the destination node:
scp root@10.10.10.2:/var/lib/libvirt/images/rntp.qcow2  /var/lib/libvirt/images/
scp root@10.10.10.2:/root/rntp.xml  /etc/libvirt/qemu/
scp root@10.10.10.2:/root/rntp.md5  /root/

Verify md5 checksum (before that change location in the rntp.md5 file):
md5sum -c rntp.md5
/root/rntp.qcow2: OK
/root/rntp.xml: OK

Copying an image to a physical device:
The diskimage will need to be in raw format.
qemu-img convert -O raw rntp.qcow2 rntp.raw
Then you just dd it onto the hard drive.
dd if=rntp.raw of=/dev/agrp-c01n01_vg0/vm01-rntp_0
Or, let qemu-img directly write onto the drive in one command:
qemu-img convert -O raw rntp.qcow2 /dev/agrp-c01n01_vg0/vm01-rntp_0

Adapting the configuration file
Old situation:

     <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/rntp.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

New situation:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/agrp-c01n01_vg0/vm01-rntp_0'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

After this step, you should be able to start your virtual machine again. If the machine boots successfully, you can remove the old disk file (I suggest to stop machine on the previous location and then test new VM for a week and only after that delete old image file).

If you move virtual machine from one machine to the other (both are not members of the same cluster) - use virsh define vm01-rntp.xml to create machine on the new location and then stop VM on the old location (virsh shutdown vm01-rntp) and disable VM autostart (rm /etc/libvirt/qemu/autostart/rntp.xml)

PS if you want to move virtual machine from CentOS 6 (libvirt 0.10.2) to CentOS 7 (libvirt 3.2.0) - xml file will not fit. Instead of using xml file - take any xml-dump file of virtual machine existing on CentOS 7 and replace needed xml attributes (<uuid> / <name> / <memory unit= / <currentMemory unit= / <vcpu placement= / <os> / <disk /  <interface ). 

Thursday, May 3, 2018

How many vCPU per pCPU.

virsh # nodeinfo
CPU model:           x86_64
CPU(s):              12
CPU frequency:       1200 MHz
CPU socket(s):       1
Core(s) per socket:  6
Thread(s) per core:  2
NUMA cell(s):        1
Memory size:         16741656 KiB


  • Socket - physical socket which the processor package sits in, on the motherboard
  • Processor package is what you get when you buy a single hardware processor.
  • Core - is a hardware term that describes the number of independent central processing units in a single computing component (Cores count = Sockets*Cores per socket)). vCPU count equals to the cores count
  • Thread - A Thread, or thread of execution, is a software term for the basic ordered sequence of instructions that can be passed through or processed by a single CPU core.
  • NUMA - Non-uniform memory access is a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. Under NUMA, a processor can access its own local memory faster than non-local memory (memory local to another processor or memory shared between processors). The benefits of NUMA are limited to particular workloads, notably on servers where the data is often associated strongly with certain tasks or users
  • ht Flag - Intel® Hyper-Threading Technology (Intel® HT Technology) delivers two processing threads per physical core - instructions are processed through two threads simultaneously,. Highly threaded applications can get more work done in parallel, completing tasks sooner. (also we see that Thread(s) per core equals to 2 - meaning that HT is enabled). Hyper-Threading doesn't double the performance, max performance increases are up to max 30%
  • CPU - equals Cores*Threads per core (in example case 6*2=12). vCPU count equals to CPU count

KVM uses Virtual CPU (vCPU) notion while assigning Processor CPU (pCPU) to the newly created virtual machine. vCPU is neither an OS thread nor a process.
Intel VT-x proposed a new mode methodology with two modes: VMX root mode and VMX non-root mode (VMX - Virtual Machine Extensions, this is set of CPU instructions added by Intel to their processors to support virtualization), for running host VMM (Virtual Machine Monitor=Hypervisor) and guest respectively.
Intel VT-x also contains a new structure: VMCS (Virtual Machine Control Structure - supports nested virtualization - a VM inside the VM ), which saves all information both host and guest need. VMCS is one per guest.
The guest code is running directly on CPU in VMX non-root mode. No software emulation layer for vCPU is needed. That’s why KVM has better performance, and there is no specific thread for guest.

A vCPU equates to 1 physical core, but (by default - rhel.link) when your VM attempts to process something, it can potentially run on any of the cores that happen to be available at that moment. The scheduler handles this, and the VM is not aware of it. You can assign multiple vCPUs to a VM which allows it to run concurrently across several cores. Cores are shared between all VMs as needed, so you could have a 4-core system, and 10 VMs running on it with 2 vCPUs assigned to each. VMs share all the cores in your system quite efficiently as determined by the scheduler. This is one of the main benefits of virtualization - making the most use of under-subscribed resources to power multiple OS instances.

The exact amount of CPU overcommitment a KVM host can accommodate will depend on the VMs and the applications they are running. A general guide for performance of {allocated vCPUs}:{total vCPU} from the Best Practices recommendations is:

  • 1:1 to 3:1 is no problem (the general recommendation is only using 1 vCPU per VM until you determine there is a need to add more than that. Over allocations of vCPU can cause lags in the VMs because too many are trying to use the same physical processors simultaneously. )
  • 3:1 to 5:1 may begin to cause performance degradation
  • 6:1 or greater is often going to cause a problem
So that if you will use overcommitment rate 3:1, then 12 vCPU (as in example) can be assigned as 36 vCPUs 

Friday, April 27, 2018

Cluster 22. Testing Live-Migration, Overall Recovery and Fail-over

Setup Descripted

We have installed 4 VMs:
  • agrp-c01n01 is the primary node for vm01 & vm02 so these VMs may start on agrp-c01n02 only when agrp-c01n01 is unavailable
  • agrp-c01n02 is the primary node for vm03 & vm04 so these VMs may start on agrp-c01n01 only when agrp-c01n02 is unavailable

VM Live Migration tests (manual, on cluster stop, manual withdrawal)

Whenever you use a migrate command (pcs resource move), Pacemaker creates a permanent location constraint pinning the resource on that node. Something like:
pcs constraint --full | grep -E "Enabled.+vm[0-9]"
    Enabled on: agrp-c01n01 (score:INFINITY) (role: Started) (id:cli-prefer-vm02-www)
This is usually undesirable, to revoke this constraint once the resource migration has been completed or when node is restored issue the pcs recource clear vm02-www. 
The resource-stickiness will cause the resource to stay where it is, anyway. But by default:
pcs property list --defaults | grep stick
 default-resource-stickiness: 0
In this posts I prefer VM to automatically migrate to the primary node when it's available. If it is pointless for you, use resource-stickiness:
pcs resource update vm02-www meta resource-stickiness=INFINITY

Manual Live Migration

pcs status resources
virsh console vm02-www # from node agrp-c01n01
uptime # note the uptime
pcs resource move vm02-www agrp-c01n02 # moving vm02-www resource from agrp-c01n01 to the agrp-c01n02
virsh console vm02-www # from node agrp-c01n02
uptime # uptime must be equal to the previous result or more
pcs constraint --full | grep -E "Enabled.+Started.+vm[0-9]"
pcs recource clear vm02-www # from any node - this should cause vm02-www to migrate back to the agrp-c01n01

Automatic Live Migration on cluster stop (on one node)

pcs status resources
virsh console vm02-www # from node agrp-c01n01
uptime # note the uptime
pcs cluster stop # stopping cluster on agrp-c01n01
pcs status # now vm01 , vm02, vm03 and vm04 must be started on agrp-c01n02
virsh console vm02-www # from node agrp-c01n02
uptime # uptime must be equal to the previous result or more
pcs constraint --full | grep -E "Enabled.+vm[0-9]" # as you see no constraints are added, because migration is caused by cluster system itself due to agrp-c01n01 going offline
pcs cluster start # starting cluster on agrp-c01n01
pcs status # now vm01 and vm02 should migrate back to agrp-c01n01 automatically

Controlled Migration and Node Withdrowal

This steps must be repeated one time for each node (first agrp-c01n01 & then agrp-c01n02):
agrp-c01n01 withdrawal:
resources=$(pcs resource | grep -E  "VirtualDomain.+n01" | awk '{print $1}')
for resource in $resources; do pcs resource move $resource agrp-c01n02; done
pcs status
pcs cluster stop # on agrp-c01n01
systemctl poweroff # on agrp-c01n01
resources=$(pcs constraint --full | grep -E "Enabled.+Started.+vm[0-9]" | awk '{print $7}' | cut -d\- -f3,4 | cut -d\) -f 1)
for resource in $resources; do pcs resource clear $resource; done
power-on power-offed node
pcs cluster start # all VM should migrate to their original positions

agrp-c01n02 withdrawal:
resources=$(pcs resource | grep -E  "VirtualDomain.+n02" | awk '{print $1}')
for resource in $resources; do pcs resource move $resource agrp-c01n01; done
pcs status
pcs cluster stop # on agrp-c01n02
systemctl poweroff # on agrp-c01n02
resources=$(pcs constraint --full | grep -E "Enabled.+Started.+vm[0-9]" | awk '{print $7}' | cut -d\- -f3,4 | cut -d\) -f 1)
for resource in $resources; do pcs resource clear $resource; done
power-on power-offed node
pcs cluster start # all VM should migrate to their original positions

VM resource restarting test

This process should be repeated for all VM's on it's primary node. Here only steps for vm01-nagios are shown but they are identical for the other VMs:
clear; tail -f -n 0 /var/log/messages
virsh console vm01-nagios
shutdown -h 0
In the output of the /var/log/messages you we'll see below lines:
....
Apr 26 16:08:16 agrp-c01n01 systemd-machined: Machine qemu-3-vm01-nagios terminated
Apr 26 16:08:17 agrp-c01n01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port vnet0
Apr 26 16:08:24 agrp-c01n01 pengine[2795]: warning: Processing failed op monitor for vm01-nagios on agrp-c01n01: not running (7)
...
Apr 26 16:08:24 agrp-c01n01 crmd[2796]:  notice: Initiating start operation vm01-nagios_start_0 locally on agrp-c01n01
...
Apr 26 16:08:25 agrp-c01n01 crmd[2796]:  notice: Result of start operation for vm01-nagios on agrp-c01n01: 0 (ok)
pcs status # vm01-nagios should be "Started"

Nodes Crash-Test

Simulating Software Crash

Crashing agrp-c01n01:
clear; tail -f -n 0 /var/log/messages # on agrp-c01n02
echo c > /proc/sysrq-trigger # on agrp-c01n01
What is in the log (key points):
16:28:17 agrp-c01n02 corosync[2217]: [TOTEM ] A processor failed, forming new configuration.
16:28:18 agrp-c01n02 attrd[2228]:  notice: Node agrp-c01n01 state is now lost
16:28:18 agrp-c01n02 kernel: dlm: closing connection to node 1
agrp-c01n02 corosync[2217]: [MAIN  ] Completed service synchronization, ready to provide service.
16:28:18 agrp-c01n02 dlm_controld[3148]: 2203 fence request 1 pid 22241 nodedown time 1524745698 fence_all dlm_stonith
agrp-c01n02 stonith-ng[2226]:  notice: Requesting peer fencing (reboot) of agrp-c01n01
16:28:22 agrp-c01n02 kernel: drbd r0: PingAck did not arrive in time.
16:28:22 agrp-c01n02 kernel: drbd r0: helper command: /sbin/drbdadm fence-peer r0
16:28:23 agrp-c01n02 pengine[2229]: warning: Cluster node agrp-c01n01 will be fenced: peer is no longer part of the cluster
16:28:23 agrp-c01n02 pengine[2229]: warning: Node agrp-c01n01 is unclean
16:28:23 agrp-c01n02 crmd[2230]:  notice: Requesting fencing (reboot) of node agrp-c01n01
16:28:26 agrp-c01n02 kernel: drbd r1: PingAck did not arrive in time.
16:28:2d6 agrp-c01n02 kernel: drbd r1: helper command: /sbin/drbdadm fence-peer r1
16:28:47 agrp-c01n02 stonith-ng[2226]:  notice: Call to fence_ipmi_n01 for 'agrp-c01n01 reboot' on behalf of stonith-api.22241@agrp-c01n02: OK (0)
16:28:47 agrp-c01n02 crmd[2230]:  notice: Peer agrp-c01n01 was terminated (reboot) by agrp-c01n02 for agrp-c01n02: OK (ref=47de2c75-7e7c-49d3-9796-17779e46e0bd) by client crmd.2230
16:28:47 agrp-c01n02 pengine[2229]:  notice:  * Start      vm02-www          (       agrp-c01n02 )
16:28:47 agrp-c01n02 pengine[2229]:  notice:  * Start      vm01-nagios        (       agrp-c01n02 )
16:28:48 agrp-c01n02 crm-fence-peer.sh[22390]: INFO peer is fenced, my disk is UpToDate: placed constraint 'drbd-fence-by-handler-r1-ms_drbd_r1'
26 16:28:48 agrp-c01n02 crm-fence-peer.sh[22271]: INFO peer is fenced, my disk is UpToDate: placed constraint 'drbd-fence-by-handler-r0-ms_drbd_r0'
16:28:49 agrp-c01n02 kernel: GFS2: fsid=agrp-c01:shared.1: jid=0: Looking at journal...
16:28:49 agrp-c01n02 kernel: GFS2: fsid=agrp-c01:shared.1: recover generation 9 done
16:28:49 agrp-c01n02 crmd[2230]:  notice: Result of start operation for vm02-www on agrp-c01n02: 0 (ok)
26 16:28:49 agrp-c01n02 crmd[2230]:  notice: Result of start operation for vm01-nagios on agrp-c01n02: 0 (ok)

BE CAREFUL: on production system it's better to start cluster on restored node, wait until r0 is UpToDate an then delete r0 constraint (r0 will be promoted to master for both nodes) and then wait until r1 UpToDate an then delete r1 constraint (r1 will be promoted to master for both nodes):
constraints=$(pcs constraint --full | grep -E "drbd-fence.+rule" | awk '{print $4}' | cut -d\: -f 2 | cut -d\) -f 1)
for constraint in $constraints; do pcs constraint remove $constraint; done
Login to the agrp-c01n01:
pcs cluster start # vm01 & vm02 will migrate to the agrp-c01n01


Crashing agrp-c01n02:
clear; tail -f -n 0 /var/log/messages # on agrp-c01n01
echo c > /proc/sysrq-trigger # on agrp-c01n02
What is in the log (key points): 
mostly the same as for agrp-c01n01

BE CAREFUL: on production system it's better to start cluster on restored node, wait until r0 is UpToDate an then delete r0 constraint (r0 will be promoted to master for both nodes) and then wait until r1 UpToDate an then delete r1 constraint (r1 will be promoted to master for both nodes):
constraints=$(pcs constraint --full | grep -E "drbd-fence.+rule" | awk '{print $4}' | cut -d\: -f 2 | cut -d\) -f 1)
for constraint in $constraints; do pcs constraint remove $constraint; done
Login to the agrp-c01n02:
pcs cluster start # vm03 & vm04 will migrate to the agrp-c01n02

Simulating Hardware Crash

Crashing agrp-c01n01:
clear; tail -f -n 0 /var/log/messages # on agrp-c01n02
power-off the node (pull power cord out of PSU)
What is in the log (key points): 
mostly the same as for agrp-c01n01 (while Simulating Software Crash), different points:
18:05:58 agrp-c01n02 corosync[4176]: [TOTEM ] A processor failed, forming new configuration.
18:06:34 agrp-c01n02 fence_ipmilan: Connection timed out
18:06:02 agrp-c01n02 crm-fence-peer.sh[6507]: No messages received in 3 seconds.. aborting
18:07:10 agrp-c01n02 stonith-ng[4190]:  notice: Call to fence_ipmi_n01 for 'agrp-c01n01 reboot' on behalf of stonith-api.6572@agrp-c01n02: Connection timed out (-110)
Apr 26 18:07:10 agrp-c01n02 stonith-ng[4190]: warning: Agent 'fence_ifmib' does not advertise support for 'reboot', performing 'off' action instead
18:07:28 agrp-c01n02 crm-fence-peer.sh[6507]: INFO peer is not reachable, my disk is UpToDate: placed constraint 'drbd-fence-by-handler-r1-ms_drbd_r1'
18:07:28 agrp-c01n02 kernel: drbd r1: helper command: /sbin/drbdadm fence-peer r1 exit code 5 (0x500)
18:07:28 agrp-c01n02 kernel: drbd r1: fence-peer helper returned 5 (peer is unreachable, assumed to be dead)
18:07:30 agrp-c01n02 stonith-ng[4190]:  notice: Call to fence_ifmib_n01 for 'agrp-c01n01 reboot' on behalf of stonith-api.6572@agrp-c01n02: OK (0)
18:07:30 agrp-c01n02 kernel: drbd r0: fence-peer helper returned 7 (peer was stonithed)
18:07:32 agrp-c01n02 crmd[4194]:  notice: Result of start operation for vm01-nagios on agrp-c01n02: 0 (ok)
18:07:32 agrp-c01n02 crmd[4194]:  notice: Result of start operation for vm02-www on agrp-c01n02: 0 (ok)
power-on agrp-c01n01
fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel2 --action status
fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel2 --action on
fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel2 --action status
Login to the agrp-c01n01 and verify it's operational
pcs stonith cleanup # on agrp-c01n02

BE CAREFUL: on production system it's better to start cluster on restored node, wait until r0 is UpToDate an then delete r0 constraint (r0 will be promoted to master for both nodes) and then wait until r1 UpToDate an then delete r1 constraint (r1 will be promoted to master for both nodes):
constraints=$(pcs constraint --full | grep -E "drbd-fence.+rule" | awk '{print $4}' | cut -d\: -f 2 | cut -d\) -f 1)
for constraint in $constraints; do pcs constraint remove $constraint; done
Login to the agrp-c01n01:
pcs cluster start # vm01 & vm02 will migrate to the agrp-c01n01
pcs status


Crashing agrp-c01n02:
clear; tail -f -n 0 /var/log/messages # on agrp-c01n02
power-off the node (pull power cord out of PSU)
What is in the log (key points): 
mostly the same as for agrp-c01n01 (while Simulating Software & Hardware Crash)


power-on agrp-c01n02
fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel3 --action status
fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel3 --action on
fence_ifmib --ip agrp-stack01 --community agrp-c01-community --plug Port-channel3 --action status
Login to the agrp-c01n02 and verify it's operational
pcs stonith cleanup # on agrp-c01n01

BE CAREFUL: on production system it's better to start cluster on restored node, wait until r0 is UpToDate an then delete r0 constraint (r0 will be promoted to master for both nodes) and then wait until r1 UpToDate an then delete r1 constraint (r1 will be promoted to master for both nodes):
constraints=$(pcs constraint --full | grep -E "drbd-fence.+rule" | awk '{print $4}' | cut -d\: -f 2 | cut -d\) -f 1)
for constraint in $constraints; do pcs constraint remove $constraint; done
Login to the agrp-c01n02:
pcs cluster start # vm01 & vm02 will migrate to the agrp-c01n01
pcs status


Administrative modes: standby, unmanaged, maintenance

Recurring monitor operations behave differently under various administrative settings:
  1. When a resource is unmanaged: No monitors will be stopped. If the unmanaged resource is stopped on a node where the cluster thinks it should be running, the cluster will detect and report that it is not, but it will not consider the monitor failed, and will not try to start the resource until it is managed again. Starting the unmanaged resource on a different node is strongly discouraged and will at least cause the cluster to consider the resource failed, and may require the resource’s target-role to be set to Stopped then Started to be recovered.
  2. When a node is put into standby: All resources will be moved away from the node, and all monitor operations will be stopped on the node, except those with role=Stopped. Monitor operations with role=Stopped will be started on the node if appropriate.
  3. When the cluster is put into maintenance mode: All resources will be marked as unmanaged. All monitor operations will be stopped, except those with role=Stopped. As with single unmanaged resources, starting a resource on a node other than where the cluster expects it to be will cause problems.
Maintenance mode and making resource unmanaged are preferred method if you are doing the online changes on the cluster nodes.  Standby mode is preferred if you need some hardware maintenance.

Managed/unmanaged resources

To make resource unmanaged via :
pcs resource unmanage libvirtd # it the same as: pcs resource update libvirtd meta is-managed=false
pcs status
 Clone Set: libvirtd-clone [libvirtd]
     libvirtd (systemd:libvirtd): Started agrp-c01n01 (unmanaged)
     libvirtd (systemd:libvirtd): Started agrp-c01n02 (unmanaged)
To make resource managed again:
pcs resource manage libvirtd # it the same as: pcs resource update libvirtd meta is-managed=true
pcs status
 Clone Set: libvirtd-clone [libvirtd]
     Started: [ agrp-c01n01 agrp-c01n02 ]

Standby/Unstandby 


Standby means that node is not permitted to run any resources but participates in voting (if note shutdown).

To move node to the standby mode:
pcs node standby agrp-c01n02
Aftre issuing - all VMs are migrated to the agrp-c01n01 and then all resources on agrp-c10n02 are stopped and node itself is shown as:
Node agrp-c01n02: standby
pcs quorum status # we'll see that standby node is also participating in voting
Node in standby mode can be restarted or shut-downed, then the status will change into:
Node agrp-c01n02: OFFLINE (standby)
Also total votes will be "1":
pcs quorum status

To clear standby mode after reboot or shutdown:
pcs cluster start # on node agrp-c01n02
pcs node unstandby agrp-c01n02


Maintenance

In a Pacemaker cluster, as in a standalone system, operators must complete maintenance tasks such as software upgrades and configuration changes. Here's what you need to keep Pacemaker's built-in monitoring features from creating unwanted side effects.
With clone and master-slave resources better way is to place node into standby mode, because node in maintenance mode can be fenced by the other node (i.e. DRBD PingAck will not arrive int time and node will be fenced).

Maintenance entire cluster:
pcs property list --defaults | grep mainte
pcs property set maintenance-mode=true
or
pcs node maintenace --all
pcs status
              *** Resource management is DISABLED ***
  The cluster will not attempt to start, stop or recover services

Online: [ agrp-c01n01 agrp-c01n02 ]

Full list of resources:
All resources are shown with " (unmanaged)" added to the end of the resource status line. 

In maintenance mode, you can stop or restart cluster resources at will. Pacemaker will not attempt to restart them. All resources automatically become unmanaged, that is, Pacemaker will cease monitoring them and hence be oblivious about their status. You can even stop all Pacemaker services on a node, and all the daemons and processes originally started as Pacemaker managed cluster resources will continue to run.
You should know that when you start Pacemaker services on a node while the cluster in maintenance mode, Pacemaker will initiate a single one-shot monitor operation (a "probe") for every resource just so it has an understanding of what resources are currently running on that node. It will, however, take no further action other than determining the resources' status.
Maintenance mode is something you enable before running other maintenance actions, not when you're already half-way through them. And unless you're very well versed in the interdependencies of resources running on the cluster you're working on, it's usually the very safest option. In short: when doing maintenance on your Pacemaker cluster, by default, enable maintenance mode before you start, and disable it after you're done.

To unmaintenance entire cluster:
pcs property set maintenance-mode=false
or
pcs node unmaintenance --all

Maintenance single node:
pcs node maintenance # will set local node into maintenance mode
pcs node maintenance agrp-c01n01 # will set agrp-c01n01 into maintenance mode


Resources disable/enable

If you need to disable (stop) resource:
pcs resource stop vm01-nagios
pcs status
 vm01-nagios (ocf::heartbeat:VirtualDomain): Started agrp-c01n01 (disabled)
virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     vm02-www                       running

To enable resource:
pcs resource enable vm01-nagios
pcs status
 vm01-nagios (ocf::heartbeat:VirtualDomain): Started agrp-c01n01
virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     vm02-www                     running
 2     vm01-nagios                    running


This tutorials were used to understand and setup clustering: 
AN!Cluster
unixarena
clusterlabs.org
hastexo.com

Monday, April 23, 2018

Cluster 21. Making installed VM (Virtual Machine) a cluster resource.

Related to libvirtd

In order to start VM as HA resource, libvirtd must be up and running (on both nodes):
creare resource make it clone and start after shredfs because libvirtd uses pools (virsh pool-list virsh pool-info) and files pool is /shared/files
pcs resource create libvirtd systemd:libvirtd
pcs resource clone libvirtd clone-max=2 clone-node-max=1 interleave=true
pcs constraint order start sharedfs-clone then start libvirtd-clone
pcs constraint colocation add libvirtd-clone with sharedfs-clone
some options can be found here: Cluster 17

Firewall setup to support KVM Live Migration

Setup firewall ports for KVM live-migration (on both nodes):
On node1:
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.10.53.2/32" port protocol="tcp" port="49152-49216" accept'
firewall-cmd --reload
firewall-cmd --list-all
On node2:
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.10.53.1/32" port protocol="tcp" port="49152-49216" accept'
firewall-cmd --reload
firewall-cmd --list-all


49152-49216 is a pool of tcp ports used randomly by virsh to perfor live migration.

Related to VM itself


In order for the cluster to manage a server, it must know where to find the "definition" file that describes the virtual machine and its hardware. When the server was created with virt-install, it saved this definition file in /etc/libvirt/qemu/
Normal libvirtd tools are not cluster-aware, so we don't want them to see our server except when it is running. We will get this done via "undefine" our VM.

First we'll share definition:
virsh list --all # list running and power-off VMs
virsh dumpxml vm02-www # view VM definition xml dump
mkdir /shared/definitions
vursh shutdown vm02-www
virsh dumpxml vm02-www > /shared/definitions/vm02-www.xml # save dump to the shared directory, this file will be used to start, stop, recover and migrate the VM
verify that xml is saved properly # because next step will destroy the VM

Stop and destroy VM:
virsh destroy vm02-www
virsh undefine vm02-www
virsh list --all # be sure that needed VM is undefined

Setup VM cluster resource (this command is executed on VM primary node - vm02-www primary node is agrp-c01n01):
pcs resource create vm02-www ocf:heartbeat:VirtualDomain hypervisor="qemu:///system" config="/shared/definitions/vm02-www.xml" migration_transport=ssh meta allow-migrate=true op monitor interval="30" timeout="30s" op start interval="0" timeout="240s" op stop interval="0" timeout="120s"

Options described(for all options see pcs resource describe VirtualDomain or man ocf_heartbeat_VirtualDomain):
  1. hypervisor="qemu:///system" - you can find this uri by executing virsh --quiet uri
  2. migration_transport=ssh - use ssh while migrating VM
  3. meta allow-migrate=true - Resources have two types of options: meta-attributes and instance attributes. Meta-attributes apply to any type of resource, while instance attributes are specific to each resource agent. Visit clusterlabs.org/meta

pcs constraint order start libvirtd-clone then vm02-www
pcs constraint colocation add vm02-www with libvirtd-clone 
Adding below constraint is needed because without it after node returning (after fail or manual cluster stop-starting) pacemaker will try to migrate VM to the primary node without waiting for DRBD promotion:
pcs constraint colocation add vm02-www with master ms_drbd_r0

Scores are calculated per resource and node. Any node with a negative score for a resource can’t run
that resource. The cluster places a resource on the node with the highest score for it. Positive values indicate a preference for running the affected resource(s) on this node — the higher the value, the stronger the preference. Negative values indicate the resource(s) should avoid this node (a value of - INFINITY changes "should" to "must"):
pcs constraint location add lc_vm02_n01 vm02-www agrp-c01n01 1
pcs constraint location add lc_vm02_n02 vm02-www agrp-c01n02 0
Above location constraints are needed to automatically live-migrate to the node which is primary for that VM. For vm02-www primary node is agrp-c01n01)

To view current score for the resource:
crm_simulate -sL | grep " vm[0-9]"
 vm02-www (ocf::heartbeat:VirtualDomain): Started agrp-c01n02
native_color: vm02-www allocation score on agrp-c01n01: -INFINITY
native_color: vm02-www allocation score on agrp-c01n02: 0
-INFINITY for agrp-c01n01 is really not because of constraint but because of agrp-c01n01 is offline

SELinux related issues

SELinux is preventing /usr/bin/virsh from read access on the file vm01-nagios.xml

semodule -DB # enables complete logging SELinux messages to the audit.log
open one more ssh to the node
tail -f -n0 /var/log/audit/audit.log
pcs resources cleanup
The message appeared:
type=AVC msg=audit(1524726228.964:514): avc:  denied  { read } for  pid=8711 comm="virsh" name="vm01-nagios.xml" dev="dm-5" ino=3477882 scontext=system_u:system_r:virsh_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file

Its's complaining about access to the vm01-nagios.xml file which is on device dm-5 and having inode 3477882. Let's find which device is it:
ls -lah /dev/mapper | grep dm-5 # It's agrp--c01n01_vg0-shared
Let's find what is inode 3477882:
find /shared -inum 3477882 # It's /shared/definitions/vm01-nagios.xml
Let's view SELinux context for /shared (we can also view context for only that file but we know that we can have many definitions in the /shared):
ls -laZ /shared # . (meaning current directory - /shared) context is system_u:object_r:unlabeled_t:s0 this context is not permissive enough, so we'll change it (only on one node but verify on the other):
semanage fcontext -a -t virt_etc_t '/shared(/.*)?' 
restorecon -r /shared 
ls -laZ /shared
semodule -B # disable audit.log for contexts with dontaudit option enabled

This tutorials were used to understand and setup clustering: 
AN!Cluster
unixarena
clusterlabs.org
rarforge.com

Restoring SQL queries from /var/log/asterisk/messages log-files.

One of customers had problem with Asterisk CEL due to insufficient HDD space left on Asterisk server. To extract SQL queries from messages, I used these sequence:
  1. cd /var/log/asterisk/
  2. grep "cel_odbc.c: Insert failed on" messages | awk '{$1=$2=$3=$4=$5=$6=$7=$8=$9=$10=$11=""; print $0}'  > messages_last
  3. removing all sequences of more than 1 white-spaces (previous command created 11 whitespace sequence):
    1. tr -s " " < messages_last > messages_truncated
  4. remove one white space left before INSERT statement:
    1. sed -i 's/^ *//' messages_truncated
  5. append ";" to the end of each line:
    1. sed -i 's/$/;/' messages_truncated
  6. execute queries from the file:
    1. mysql -u root -p asterisk
    2. SOURCE /var/log/asterisk/messages_truncated
    3. or
    4. mysql -u root -p asterisk < /var/log/asterisk/messages_truncated