Cluster 7. What is split-brain, quorum, DLM & fencing, totem protocol & CPG.

Split-brain

A split-brain is a state in which nodes lose contact with each other and then try to take control of shared resources or simultaneously provide clustered services. This leads to actually corrupting and loosing data. To avoid split-brain situations quorum is used.

Quorum

Quorum algorithm used in the Red Hat Cluster is a simple majority meaning that more than half of the hosts must be online and communicating in order to provide services: (nodes_count / 2 + 1) rounding down:

• If we have 3 nodes in a cluster, voices count = 3, quorum  = 3 / 2 + 1 = 1.5 + 1 = 2.5 ~ 2 , so at least 2 nodes needed to form new cluster after 1 node fails, if 2 nodes fail cluster will hung
• If we have 4 nodes in a cluster, voices count = 4, quorum = 4 / 2 + 1 = 2 + 1 = 3 , so at least 3 nodes needed to form new cluster after 1 node fails, if 2 nodes fail cluster will hang
• If we have 5 nodes in a cluster, voices count = 5, quorum = 5 / 2 + 1 = 2.5 + 1 = 3.5 ~ 3 , so at least 3 nodes needed to form new cluster after 1 node fails, if 2 nodes fail cluster will rebuild

In cluster with 2 nodes any failure will cause 50/50 split, hanging both nodes. To make 2 nodes cluster fault-tolerant fencing is used (in corosync 2.4 we have options to use quorum with 2 node cluster but fencing also needed).

If cluster is split into two or more partitions, group of machines having quorum, can form new cluster.

PS we can use qdisk to form quorum in cluster of 2 nodes, but this one is not working with DRBD which we are going to use for HDD replication. Also we are going to use corosync 2.4 which has options like two_node & wait_for_all which are not working with qdisk.

Fencing aka STONITH

Fencing means putting the target node into a state where it can not affect cluster resources or provide cluster services. This can be accomplished by powering it off (power fencing), disconnect it from SAN storage and/or network (fabric fencing).

Fence is absolutely critical part for clustering. Without fully functional fencing your cluster will fail. 

Linux-HA used STONITH ("Shoot The Other Node In The Head") term and Red Hat used the term - "fencing". Both terms can be used interchangeably.
When nodes fail or cluster split into partitions winning node or partition (winning here means - "having quorum") will fence losers (in two node cluster with corosync 2.4 one node will have quorum and try to fence the other node, with network failure this can end with fencing loop - both nodes fencing each other forever. To solve that - you need to setup delay in fencing for the preferred node).
If all (or the only) configured fence fails, fence daemon will start over. Fence daemon will wait and loop forever until a fence agent succeeds. During this time, the cluster is effectively hung.
Once a fence_agent succeeds, fence daemon notifies DLM and lost locks are recovered. This is how Fencing & DLM are cooperating.

DLM (Distributed Lock Manager)

File system locking in Linux is done by POSIX or other type of locks available in system. DLM is used by cluster storage and resource manager in order to organize and serialize the access (it manages locks). dlm daemon runs in user-space (kernel space is used to run OS critical components and user-space is used to provide memory for software), this software communicates with DLM in kernel. The lockspace  (locking definite resource) is given to the requester node,  the other node can request lockspace only after first node releases the lock.
PS - DLM is used only with cluster aware file-systems.

Totem protocol, CPG & virtual synchrony

Totem protocol is used to send token messages between cluster nodes. A token is passed around to each node, the node does some work, and then it passes the token on to the next node. This goes around and around all the time. Should a node not pass its token on after a short time-out period (defaults to 238ms), the token is declared lost, an error count (defaults to 4 losses) goes up and a new token is sent. If too many tokens are lost in a row, the node is declared lost. The cluster checks which members it still has, and if that provides enough votes for quorum.

The closed process group (CPG) is a small process layer on top of the totem protocol provided by corosync. It handles the sending and delivery of messages among nodes in a consistent order. It adds PIDs and group names to the membership layer. Only members of the group get the messages, thus it is a "closed" group. So in other words - CPG is simply a private group of processes in a cluster.
The ordered delivery of messages among cluster nodes is referred to as "virtual synchrony". 

Virtual synchrony (DLM & CPG cooperation)

DLM messages are in ordered delivery due to using totem's CPG. When a node wants to start a clustered service (cluster-aware file-system), this node can start this service only after achieving a lock from DLM. After starting this clustered service, this node announces other nodes - members of the CPG. So after issuing DLM (when stating clustered service or requesting storage lock etc.) every member (node) notifies other CPG members (nodes). 

Messages can only be sent to the members of the CPG while the node has a totem token from corosync.

This tutorial was used to understand and setup clustering: AN!Cluster

Cluster 6. Putting all devices hostnames and IP addresses to the /etc/hosts

Three different networks are used in our cluster:

1. BCN (Back-Channel Network - 10.0.53.nodeIP/24) for cluster management traffic, IPMI, switches - node names will be as  agrp-c01n01.bcn
1. IPMI IP will be - 10.clusterSerialNumber*10.53.nodeIP*10+1/24 (i.e. 10.10.53.11)
2. switch stack Ip will be - 10.clusterSerialNumber*10.53.nodeIP*10+2/24 (i.e. 10.10.53.12):
1. access stack and execute:
2. int vlan 100
3. ip address 10.10.53.12 255.255.255.0
4. ip address 10.10.53.22 255.255.255.0 secondary
5. do sh int vlan 100
2. SN (Storage Network - 10.0.52.nodeIP/24) - node names will be as - agrp-c01n01.sn
3. IFN (Internet-Facing Network - 10.0.51.ServerIP/24) - only for servers (virtual servers, hosted on a node) - node names will be as - agrp-c01n01.ifn

connect IPMI iLOes agrp-c01n01 goes to 1/0/17 and agrp-c01n02 to 2/0/17 - Back-Channel Network - 10.clusterSerialNumber*10.53.nodeIP/24 => 10.10.53.[12]0

Put below lines to the  /etc/hosts (it will be the same on both nodes):

### Nodes 

# agrp-c01n01

10.10.53.1    agrp-c01n01.bcn agrp-c01n01

10.10.53.11  agrp-c01n01.ipmi

10.10.52.1    agrp-c01n01.sn

172.16.51.1    agrp-c01n01.ifn

# agrp-c01n01

10.10.53.2    agrp-c01n02.bcn agrp-c01n02

10.10.53.21  agrp-c01n02.ipmi

10.10.52.2    agrp-c01n02.sn

172.16.51.2    agrp-c01n02.ifn

# Network Switches
10.10.53.12 agrp-stack01

10.10.53.22 agrp-stack01

Save and exit, verify with ping script. This script will ping every host in /etc/hosts file and then will display ping result showing how many packets are send, received and packet loss:
for name in $(grep -E "^(172|10)" /etc/hosts | awk '{print $2}');
do
   echo "NAME=$name";
   ping $name -c 3 | grep "packet loss";
   echo "";
done

Only ping to agrp-c01n01.ipmi & agrp-c01n02.ipmi must response "Destination Host Unreachable" because we haven't yer set up IPMI

This tutorial was used to understand and setup clustering: AN!Cluster

Cluster 5. Nodes naming, Configuring Interfaces, Linux bonds and bridge or Open vSwitch.

Nodes naming convention:

1. four letter code of the cluster owner name (i.e. AIST Group becomes agrp)
2. plus c+cluster number (c01 - first cluster in a company)
3. plus n+01 or 02 (node number in a cluster)
4. so the name will be: agrp-c01n01 & agrp-c01n02

Change host-names on both nodes:

• hostnamectl set-hostname agrp-c01n01 --static
• hostnamectl status
• logout
• login
• verify that hostname is displayed properly both on login screen and on CLI prompt:
• agrp-c0n01 login:
• [root@agrp-c01n01 ~]#

All commands below must be executed on both nodes (with proper IP addresses, here will be only agrp-c01n01 related commands).

Bond is the same as LAG (Link Aggregation) - RAID1 for network interfaces (if one goes down, the other will remain working).

Linux bonding and bridging 

Nodes naming and IP addresses:

Node	IP & BCN dev	IP & SN dev	Ip & IFN dev
agrp-c01n01	10.10.53.1 on bcn_bond1	10.10.52.1 on sn_bond1	172.16.51.1 on ifn_bridge1 (ifn_bond1 slaved)
agrp-c01n02	10.10.53.2 on bcn_bond1	10.10.52.2 on sn_bond1	172.16.51.2 on ifn_bridge1 (ifn_bond1 slaved)

In other articles the table was such this one:

Subnet	VID	NIC	Link 1	NIC	Link 2	Bond	Net IP
BCN	100	eno1	bcn_link1	eno4	back_link.100	bcn_bond	10.10.53.0/24
SN	200	eno2	sn_link1	eno4	back_link.200	sn_bond	10.10.52.0/24
IFN	51	eno3	ifn_link1	eno4	back_link.51	ifn_bond	172.16.51.0/24

That was so for simplicity. We will be using VLAN on all physical interfaces, so the actual table must be:

Subnet	VID	NIC	Link 1	NIC	Link 2	Bond	Net IP
BCN	100	eno1	bcn_link1.100	eno4	back_link.100	bcn_bond1	10.10.53.0/24
SN	200	eno2	sn_link1.200	eno4	back_link.200	sn_bond1	10.10.52.0/24
IFN	51	eno3	ifn_link1.51	eno4	back_link.51	ifn_bond1	172.16.51.0/24

ifn_bridge1 will be used as virtual switch for our servers (VMs) - it will give our VMs access to the VLAN 51 (our IFN). ifn_bond1 will connect to the ifn_bridge1 to connect to the real world

BCN setup

Setup back_link to be a member of  VLAN 100 (BCN):

back_link, only below lines must be in config file:

DEVICE=back_link
NAME=back_link

BOOTPROTO=none

ONBOOT=yes

HWADDR=proper_MAC_here

back_link.100

DEVICE=back_link.100

NAME=back_link.100

BOOTPROTO=none

ONBOOT=yes

VLAN=yes
SLAVE=yes
MASTER=bcn_bond1

Setup bcn_link1 to represent actual VLAN:
bcn_link1

DEVICE=bcn_link1

NAME=bcn_link1

BOOTPROTO=none

ONBOOT=yes

HWADDR=proper_MAC_here

bcn_link1.100

DEVICE=bcn_link1.100

NAME=bcn_link1.100

BOOTPROTO=none

ONBOOT=yes

VLAN=yes
SLAVE=yes
MASTER=bcn_bond1

Bonding options:

1. mode=1 => Active/Passive
2. miimon=100 => test interfaces every 100ms (MII - (Media Independent Interface) means that media type can be any - fiber,copper etc. / mon - monitoring) 
3. downdelay=0 => when link goes down immediately switch to the other interface in bond
4. updelay=120000 => switch back to the primary interface in 2 minutes
5. use_carrier=1 => check the link state

Setup bcn_bond1:

vi /etc/sysconfig/network-scripts/ifcfg-bcn_bond1

DEVICE="bcn_bond1"

BOOTPROTO="none"

ONBOOT="yes"

BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=bcn_link1.100"

IPADDR=10.10.53.1

NETMASK=255.255.255.0

systemctl restart network.service

After setting agrp-c01n02, verify ping between nodes:

agrp-c01n01# ping 10.10.53.2
agrp-c01n02# ping 10.10.53.1

SN setup

Setup back_link to be a member of  VLAN 200 (SN):

back_link.200

DEVICE=back_link.200

NAME=back_link.200

BOOTPROTO=none

ONBOOT=yes

VLAN=yes
SLAVE=yes
MASTER=sn_bond1

Setup sn_link1 to represent actual VLAN:
sn_link1

DEVICE=sn_link1

NAME=sn_link1

BOOTPROTO=none

ONBOOT=yes

HWADDR=proper_MAC_here

sn_link1.200

DEVICE=sn_link1.200

NAME=sn_link1.200

BOOTPROTO=none

ONBOOT=yes

VLAN=yes
SLAVE=yes
MASTER=sn_bond1

Setup sn_bond1:

vi /etc/sysconfig/network-scripts/ifcfg-sn_bond1

DEVICE="sn_bond1"

BOOTPROTO="none"

ONBOOT="yes"

BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=sn_link1.100"

IPADDR=10.10.52.1

NETMASK=255.255.255.0

systemctl restart network.service

After setting agrp-c01n02, verify ping between nodes:

agrp-c01n01# ping 10.10.52.2
agrp-c01n02# ping 10.10.52.1

IFN setup

Setup back_link to be a member of  VLAN 51 (IFN):

back_link.51

DEVICE=back_link.51

NAME=back_link.51

BOOTPROTO=none

ONBOOT=yes

VLAN=yes
SLAVE=yes
MASTER=ifn_bond1

Setup ifn_link1 to represent actual VLAN:
ifn_link1

DEVICE=ifn_link1

NAME=ifn_link1

BOOTPROTO=none

ONBOOT=yes

HWADDR=proper_MAC_here

ifn_link1.51

DEVICE=ifn_link1.51

NAME=ifn_link1.51

BOOTPROTO=none

ONBOOT=yes

VLAN=yes
SLAVE=yes
MASTER=ifn_bond1

Setup ifn_bond1:

vi /etc/sysconfig/network-scripts/ifcfg-ifn_bond1

DEVICE="ifn_bond1"

BOOTPROTO="none"

ONBOOT="yes"

BONDING_OPTS="mode=1 miimon=100 use_carrier=1 updelay=120000 downdelay=0 primary=ifn_link1.51"

BRIDGE=ifn_bridge1

Setup ifn_bridge1:
DEFROUTE=yes allows to use this interface as window to the outer world

vi /etc/sysconfig/network-scripts/ifcfg-ifn_bridge1

DEVICE=ifn_bridge1
TYPE=Bridge

BOOTPROTO=none

IPADDR=172.16.51.1

NETMASK=255.255.255.0
GATEWAY=172.16.51.254
DNS1=8.8.8.8
DNS2=8.8.4.4
DEFROUTE=yes

systemctl restart network.service

ping default gateway:
ping 172.16.51.254

After setting agrp-c01n02, verify ping between nodes:

agrp-c01n01# ping 172.16.51.2
agrp-c01n02# ping 172.16.51.1

Verifying

On both nodes verify master and slaves and interface states (Up/Down):
ip link | grep ifn
ip link | grep sn
ip link | grep bcn
ip link | grep back

Verify bonds (settings, slave status, failures count):
cat /proc/net/bonding/ifn_bond1
cat /proc/net/bonding/sn_bond1

cat /proc/net/bonding/bcn_bond1

Verify bridge (ifn_bridge1 must be shown, STP enabled must be no):

brctl show

PS if you encounter MAC flapping error on Cisco stack, like:
%SW_MATM-4-MACFLAP_NOTIF: Host aaaa.bbbb.cccc in vlan 51 is flapping between port Po1 and port Gi2/0/4
Then add MACADDR parameter to all bond interfaces. This MACADDR must be equal to the MAC of the non backup-link because back_link is in 3 VLANs and that can cause bond to choose back_link NIC MAC for all bond and VLAN interfaces (by default bond uses first added slave's MAC as it's own MAC).

Open vSwitch

We will bond all 4 interfaces (from eno1 through eno4) to the OvS bond ovs_bond. And then we'll create OvS internal ports and assign them IP:

Subnet	VID	OvS internal port	Net IP
BCN	100	bcn-bond1	10.10.53.0/24
SN	200	sn-bond1	10.10.52.0/24
IFN	51	ifn-bond1	172.16.51.0/24

Nodes naming and IP addresses:

Node	IP & BCN dev	IP & SN dev	Ip & IFN dev
agrp-c01n01	10.10.53.1 on bcn-bond1	10.10.52.1 on sn-bond1	172.16.51.1 on ifn-bond1
agrp-c01n02	10.10.53.2 on bcn-bond1	10.10.52.2 on sn-bond1	172.16.51.2 on ifn-bond1

Below commands must be executed on both nodes (with parameters appropriate to each node)

Create OvS bridge and bonds

Create OvS switch:

ovs-vsctl add-br ovs_kvm_bridge

Disable STP on this bridge:
ovs-vsctl set bridge ovs_kvm_bridge stp_enable=false

Add bonds to the ifn_bridge:

ovs-vsctl add-bond ovs_kvm_bridge ovs_bond eno1 eno2 eno3 eno4 trunks=100,200,51

In future if you need to add new VLANs to the trunk, execute below command with proper VLANs list:

ovs-vsctl set port ovs_bond trunks=100,200,300,400 etc.

Enabling LACP LAG protocol:

ovs-vsctl set port ovs_bond lacp=active bond_mode=balance-slb  bond-updelay=120000 bond-downdelay=0 other_config:lacp-time=fast  other_config:lacp-fallback-ab=true# no space is allowed in "config:lacp" part of configuration

To view bond interface configuration:
ovs-vsctl list Port ovs_bond

If you made mistake while configuring (i.e. wrote "lacp_time" instead of "lacp-time"):
ovs-vsctl remove port ovs_bond other_config lacp_time fast

lacp-time - either slow or fast -defines whether LACP packets are sent every 1 second, or every 30 seconds.
lacp-fallback-ab - if LACP failes - Active-Backup bonding will be used
balance-slb - Source-load Balancing (this is default on Cisco LACP bonds - sh run all | incl load-balance will give you src-mac):

1. The source MAC address is extracted, and a hashing algorithm is used to map it to a hash number 0-255. 
2. Each hash is assigned to one of the NICs on the bond, which means packets with the same hash are always sent through the same NIC. 
3. If a new hash is found, it is assigned to the NIC that currently has the lowest utilization. 
4. In practice, this means that when virtual machines (VMs) are set up on a bond, packets from one VM (with the same source MAC) will always be sent through the same NIC.

To remove ports and bridge (if something went wrong):

ovs-vsctl del-port ovs_kvm_bridge ovs_bond1

ovs-vsctl del-br ovs_kvm_bridge

To view bond configuration:

ovs-appctl bond/show ovs_bond # bond_mode must be active-backup / lacp_status = negotiated / all interfaces slave eno{1..4} : enabled
ovs-appctl lacp/show ovs_bond | head -n 7 # status: active negotiated / lacp_time: fast  

To view MAC address table:

ovs-appctl fdb/show ovs_kvm_bridge

Below command can be used to verify overall OvS bridge configuration (including STP status), -S option makes output scroll-able with keyboard left-right arrow keys:

ovsdb-client dump | less -S

Create OvS internal ports for node and assign them IP:

Setup IFN ifn-bond1, make it internal and assign VLAN ID 51:

ovs-vsctl add-port ovs_kvm_bridge ifn-bond1 -- set interface ifn-bond1 type=internal -- set port ifn-bond1 tag=51

Assign an IP to ifn-bond1:

vi /etc/sysconfig/network-scripts/ifcfg-ifn-bond1

DEVICE=ifn-bond1
NAME=ifn-bond1
ONBOOT=yes

BOOTPROTO=none

IPADDR=172.16.51.1

NETMASK=255.255.255.0
GATEWAY=172.16.51.254
DNS1=8.8.8.8
DNS2=8.8.4.4
DEFROUTE=yes

Setup BCN bcn-bond1, make it internal and assign VLAN ID 100:

ovs-vsctl add-port ovs_kvm_bridge bcn-bond1 -- set interface bcn-bond1 type=internal -- set port bcn-bond1 tag=100

vi /etc/sysconfig/network-scripts/ifcfg-bcn-bond1

DEVICE="bcn-bond1"

BOOTPROTO="none"

ONBOOT="yes"

IPADDR=10.10.53.1

NETMASK=255.255.255.0

Setup SN sn-bond1, make it internal and assign VLAN ID 200:

ovs-vsctl add-port ovs_kvm_bridge sn-bond1
ovs-vsctl set interface sn-bond1 type=internal
ovs-vsctl set port sn-bond1 tag=200

vi /etc/sysconfig/network-scripts/ifcfg-sn-bond1

DEVICE="sn-bond1"

BOOTPROTO="none"

ONBOOT="yes"

IPADDR=10.10.52.1

NETMASK=255.255.255.0

systemctl restart network.service

To list all ports which OpenvSwitch sees:
ovs-vsctl list-ports ovs_kvm_bridge # will show:
bcn-bond1
ifn-bond1
ovs_bond
sn-bond1

To listen to the ports traffic:
yum install tcpdump
tcpdump -i port_name # port name is one of the ports seen by OvS

Verifying

IFN test:
From agrp-c01n01 ping 172.16.51.2
From agrp-c01n01 ping 172.16.51.254
From agrp-c01n02 ping 172.16.51.1
From agrp-c01n01 ping 172.16.51.254
BCN test:
From agrp-c01n01 ping 10.10.53.2
From agrp-c01n02 ping 10.10.53.1
SN test:
From agrp-c01n01 ping 10.10.52.2
From agrp-c01n02 ping 10.10.52.1

After-setup steps

Backup configs after setting done (either using Linux bonding and bridging or Open vSwitch):
rsync -av /etc/sysconfig/network-scripts /root/backups

This tutorials were used to understand and setup clustering: 
AN!Cluster
brezular.com
citrix.com

Cluster 4. Mapping physical interfaces to device names

If you are going to use OvS networking, then skip this article and go to the "Cluster 5" article.

Subnet	VID	NIC	Link 1	NIC	Link 2	Bond	Net IP
BCN	100	eno1	bcn_link1	eno4	back_link.100	bcn_bond	10.10.53.0/24
SN	200	eno2	sn_link1	eno4	back_link.200	sn_bond	10.10.52.0/24
IFN	51	eno3	ifn_link1	eno4	back_link.51	ifn_bond	172.16.51.0/24

First we'll disable new NIC naming which comes with CentOS7 by default:

biosdevname=0
ln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules

Lets add MAC addresses to all ifcfg files (on the both nodes):
cd /etc/sysconfig/network-scripts

for int in $(ls -1 ifcfg-eno* | cut -d'-' -f2);

do

   mac=$(ip link show $int | grep ether | awk '{print $2}');
   echo "HWADDR=\"$mac\"" >> ifcfg-$int;
done

Now rename device names according to the table above:
mv ifcfg-eno1 ifcfg-bcn_link1
mv ifcfg-eno2 ifcfg-sn_link1
mv ifcfg-eno3 ifcfg-ifn_link1
mv ifcfg-eno4 ifcfg-back_link1

Now change NAME and DEVICE parameters in the appropriate files:
for name in bcn_link1 sn_link1 ifn_link1 back_link:
do
   sed -i "s/DEVICE=.*/DEVICE=$name/" ifcfg-$name;

   sed -i "s/NAME=.*/NAME=$name/" ifcfg-$name;
done

Now we can reboot each node and verify NIC names:
reboot
After reboot to verify proper naming, execute:
ip link

The last step will be manual test (on both nodes):

1. tail -f -n 0 /var/log/messages # -f means "as file growth" , -n 0 means "initially show no lines"
2. then unplug and plug-in cables
3. if every cable becomes "Link is down" and then "Link is up", then everything is OK 
4. backup new configs:
5. rsync -av /etc/sysconfig/network-scripts /root/backups

This tutorial was used to understand and setup clustering: AN!Cluster

Cluster 3. Network Switches.

You can use any switches that support VLAN and multicasting groups (also it's possible to use unicast at least with corosync 2.4.0). I'll use Cisco Catalyst 2960-S stackable switches (they form a stack - which can be managed as one switch).
We will use 2 Cisco stack switches (you can use just one switch) in order to make links redundant - eno4 of nodes must be connected to different switches, eno1, eno2, eno3 must be connected to the switch where the other node's eno4 interface is connected.

To setup stack:

1. Connect stack cables to the proper ports Stack1 to Stack1 and Stack2 to Stack2
2. Connect console cable to the Cisco and to your PC/NB
3. power on switches
4. verify that switch knows whe is a mater and who is a slave:
1. do sh sw
2. Master switch  Role must be - Master, current state - Ready
3. Slave switch Role must be - Member, current state  - Ready
4. Also verify that MSTR led on master switch is green
5. Verify stack-ports:
1. do sh sw stack-ports - all ports must be - OK
2. power-off master switch and verify that slave becomes master:
1. do sh sw
2. Removed switch Role must change to - Member, State must change to - Removed
3. Remaining switch Role must change to - Master, State must change to - Ready

Now you can setup stack from any switch:

1. name stack like agrp-stack01 (agrp is 4 letter owner code, stack01 is simply serial number of the stack, it's our first stack and because of that stack serial number is 01):
1. hostname agrp-stack01
2. configure username and password (simple passwords are giver for reference only):
1. aaa new-model
2. aaa authentication login default local
3. username admin privilege 15 secret 123456
4. enable secret 12345
3. create VLANs:
1. vlan 100
2. name BCN
3. vlan 200
4. name SN
5. vlan 51
6. name IFN
7. vlan 1000
8. name Deafult
4. disable all interfaces:
1. int range gi 1/0/1 - 28, gi 2/0/1-28
2. shut
3. do wr
5. enable needed interfaces and make them member of the needed VLAN, also disable STP on ports (STP requires blocking traffic to prevent loops, such a behavior can cause nodes think other node is dead while it's however alive):
1. disable STP for BCN and SN:
1. no spanning-tree vlan 100
2. no spanning-tree vlan 200
3. no spanning-tree vlan 51
2. port 1/0/24 & 2/0/24 will be IFN uplink ports - going to the other switch:
1. int ra gi 1/0/24 , gi 2/0/24
1. no shut
2. create LAG/bonding as in my case other switch is single switch, not a stack:
3. channel-group 1 mode on
2. configure Port-Channel1:
1.  int Po1
2. sw mode access
3. sw nonegotiate
4. sw access vlan 51
5. show etherchannel summary:
1. "Ports" must be - Gi1/0/24(P) Gi2/0/24(P) - meaning that both ports are bundled in a LAG
3. Configure iLO port - port 1/0/17 & 2/0/17 will be iLO interface connected ports (here we will insert node1 iLO to the switch1 and node2 iLO to the switch2):
1. node1 iLO - 1/0/17 - label this cable c01n01_ipmi
2. node2 iLO - 2/0/17 - label this cable c01n02_ipmi
3. int ra gi 1/0/17 , gi 2/0/17
1. no shut
2. sw nonegotiate
3. sw mode access
4. sw access vlan 100
4. int gi1/0/17
1. description agrp-c01n01
5. int gi2/0/17
1. description agrp-c01n02

Switch ports will be used as 5 ports per cluster, so that we can use each switch stack for serving up to 4 clusters:

• 1st cluster - gi1/0/1-4,17 & gi2/0/1-4,17
• 2nd cluster - gi1/0/5-8,19 & gi2/0/5-8,19
• 3rd cluster - gi1/0/9-12,21 & gi2/0/9-12,21
• 4th cluster - gi1/0/13-16,23 & gi2/0/13-16,23

We have two options to further setup our stack, one is for Linux bonding and bridging and the other  is for Open vSwitch.

Linux bonding and bridging

Subnet	VID	NIC	Link 1	NIC	Link 2	Bond	Net IP
BCN	100	eno1	bcn_link1	eno4	back_link.100	bcn_bond	10.10.53.0/24
SN	200	eno2	sn_link1	eno4	back_link.200	sn_bond	10.10.52.0/24
IFN	51	eno3	ifn_link1	eno4	back_link.51	ifn_bond	172.16.51.0/24

1. port 1/0/1 & 2/0/1 will be BCN ports:
1. node1 eno1 - 1/0/1
2. node2 eno1 - 2/0/1
3. int ra gi 1/0/1 , gi 2/0/1
1. no shut
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 100
5. sw trunk native vl 1000
2. port 1/0/2 & 2/0/2 will be SN ports:
1. node1 eno2 - 1/0/2
2. node2 eno2 - 2/0/2
3. int ra gi 1/0/2 , gi 2/0/2
1. no shut
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 200
5. sw trunk native vl 1000
3. port 1/0/3 & 2/0/3 will be IFN ports:
1. node1 eno3 - 1/0/3
2. node2 eno3 - 2/0/3
3. int ra gi 1/0/3 , gi 2/0/3
1. no shut
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 51
5. sw trunk native vl 1000
4. port 1/0/4 & 2/0/4 will be backup ports (here we will insert node1 port eno4 to the switch2 and node2 eno4 to the switch1):
1. node1 eno4 - 2/0/4
2. node2 eno4 - 1/0/4
3. int ra gi 1/0/4 , gi 2/0/4
1. no shut
2. sw nonegotiate
3. sw mode trunk
4. sw trunk allowed vlan 100,200,51
5. sw trunk native vl 1000

Open vSwitch

We will bond all 4 interfaces (from eno1 through eno4) to the OvS bond - ovs_bond.
And then we'll create OvS internal ports and assign them IP:

Subnet	VID	OvS internal port	Net IP
BCN	100	bcn-bond1	10.10.53.0/24
SN	200	sn-bond1	10.10.52.0/24
IFN	51	ifn-bond1	172.16.51.0/24

1. ports Gi1/0/1-1/0/4 and Gi2/0/1-2/0/4 will be trunk ports carrying all VLANs:
1. node1 eno1 and eno3 - 1/0/1 & 1/0/3 - label this 2 cables eno1_c01n01_ovs_bond and eno3_c01n01_ovs_bond
2. node1 eno2 and eno4 - 2/0/2 & 2/0/4 - label this 2 cables eno2_c01n01_ovs_bond and eno4_c01n01_ovs_bond
3. node2 eno1 and eno3 - 2/0/1 & 2/0/3 - label this 2 cables eno1_c01n02_ovs_bond and eno3_c01n02_ovs_bond
4. node2 eno2 and eno4 - 1/0/2 & 1/0/4 - label this 2 cables eno2_c01n02_ovs_bond and eno4_c01n02_ovs_bond
5. int ra gi 1/0/1, gi 1/0/3, gi 2/0/2, gi 2/0/4
1. description agrp-c01n01
2. channel-group 2 mode active #enabling LACP use different channel-group numbers for nodes
3. no shut
6. int Po2
1. description agrp-c01n01
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 100,200,51
5. sw trunk native vl 1000
6. no shut
7. int ra gi 1/0/2, gi 1/0/4, gi 2/0/1, gi 2/0/3
1. description agrp-c01n02
2. channel-group 3 mode active #enabling LACP use different channel-group numbers for nodes
3. no shut
8. int Po3
1. description agrp-c01n02
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 100,200,51
5. sw trunk native vl 1000
6. no shut
9. sh int port-channel {1|2|3} # to view info about LAG interfaces (choose needed LAG number)

LACP bandwidth - the maximum through-output will remain equal to the through-output of the single link. In fact you get more lanes to move but the maximum speed remains the same. By enabling LACP you increase maximum overall bandwidth. This achieved using load-balancing (Cisco default LB mechanism is source-MAC balancing).

This tutorial was used to understand and setup clustering: AN!Cluster 

Cluster 2. Post Install steps, Networking initial setup.

First and most important:
YOU MUST HAVE PHYSICAL ACCESS TO BOTH NODES IN ORDER TO SETUP CLUSTER

As for now we have 2 nodes (physical hardware server, which is/will be a member of a cluster) with installed OS. In this article we will do some steps needed after installation of the OS and iLO access.

1. First of all connect both servers to the network and give them Internet access permissions (don't think about IP addressing scheme, for now we just need Internet access)
2. on both nodes
1. yum update -y
2. NM makes many decisions itself which is not appropriate for cluster:
1. yum remove NetworkManager -y
3. verify that firewalld enabled and started:
1. systemctl status firewalld

Backup existing network configs (on both nodes):

mkdir -p /root/backups/
yum install rsync -y
# -v - be verbose
# -a archive-mode (recursive, copy links, preserve (permissions, timestamps, owners, groups, dev-files)
rsync -av /etc/sysconfig/network-scripts /root/backups/

Enabling all interfaces (on both nodes):

cd /etc/sysconfig/network-scripts

for int in $(ls -1 ifcfg-eno*);

do

   sed -i 's/ONBOOT=.*/ONBOOT="yes"/' $int;

   sed -i 's/BOOTPROTO=.*/BOOTPROTO="none"/' $int;

done
to check changes:

# -U0 will show anly changed line if diff

# verify all files:

for int in $(ls -1 ifcfg-eno*);

do

   diff -U0 /root/backups/network-scripts/ifcfg-$int ifcfg-$int;

done

systemctl enable network.service

systemctl start network.service
systemctl status network.service

# to verify that all interfaces are enabled:

ip link

Four different networks will be used in our cluster:

1. BCN (Back-Channel Network - 10.clusterSerialNumber*10.53.nodeIP/24) - for cluster management 
2. IPMIN (IPMI/iLO Network - 10.clusterSerialNumber*10.53.nodeIP+10+1/24) - for cluster management 
3. SN (Storage Network - 10.clusterSerialNumber*10.52.nodeIP/24) - for nodes storage replication 
4. IFN (Internet-Facing Network - 172.16.51.ServerIP/24) - for access to nodes anf for servers (virtual servers, hosted on a node)

Disable IPv6:

vi /etc/sysctl.conf
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.all.disable_ipv6 = 1

Disable zeroconf (Ip addresses starting with 169):

vi /etc/sysconfig/network

NOZEROCONF=true

reboot both servers

Two options in networking setup

We have two options for networking setup. One option is using Linux bonding and bridging, provided by kernel and bridge-utils package and the other is to use OvS (Open vSwitch) provided by OvS.
Linux bridge doesn't "understand" VLANs, it just connects VM Server virtual ports to the outer world. To support more than one VLAN with Linux bridges we need to setup as many bridges as VLANs count we desire to be.
OvS is more wide approach as in future you can easily add more VLANs to your cluster (you can serve VMs in more than one VLAN). Also we can say that OvS supports all (or many of) features normal hardware switch will support.

Linux bonding and bridging

yum install bridge-utils -y

We will be using four interfaces, bonded into three pairs of one physical NIC with VLAN and one SubNIC with VLAN in Active/Passive (mode=1 other types are not recommended for reliable clustering environment) configuration.
It's our fisrst cluster, so clusterSerialNumber = 1. IP address 2nd octet will be 1 *10 = 10

eno1 => bcn_link1

eno2 => sn_link1

eno3 => ifn_link1

eno4 => back_link.100 / back_link.200 / back_link.51

To find physical port corresponding to the CentOS links, you can use ethtool -p command, i.e.:
ethtool -p eno4  # physical port corresponding to this interface will blink until you Ctrl+C

Subnet	VID	NIC	Link 1	NIC	Link 2	Bond	Net IP
BCN	100	eno1	bcn_link1	eno4	back_link.100	bcn_bond	10.10.53.0/24
SN	200	eno2	sn_link1	eno4	back_link.200	sn_bond	10.10.52.0/24
IFN	51	eno3	ifn_link1	eno4	back_link.51	ifn_bond	172.16.51.0/24

Open vSwitch

Proceed to this link to install OvS.

We will be using four interfaces, bonded into two pairs in Active/Passive configuration.
It's our fisrst cluster, so clusterSerialNumber = 1. IP address 2nd octet will be 1 *10 = 10

eno1 => ovs_bond1

eno2 => ovs_bond2

eno3 => ovs_bond1

eno4 => ovs_bond2

To find physical port corresponding to the CentOS links, you can use ethtool -p command, i.e.:
ethtool -p eno4  # physical port corresponding to this interface will blink until you Ctrl+C

We will bond all 4 interfaces (from eno4 through eno4) to the OvS bonds. And then we'll create OvS internal ports and assign them IP:

Subnet	VID	OvS internal port	Net IP
BCN	100	bcn-bond1	10.10.53.0/24
SN	200	sn-bond1	10.10.52.0/24
IFN	51	ifn-bond1	172.16.51.0/24

This tutorial was used to understand and setup clustering: AN!Cluster

Cisco ATM / DSL setup and troubleshooting

Layers in the stack:

• Layer 1 - DSL
• Layer 2.1 - ATM
• Layer 2.2 - RFC1483B
• Layer 3 - IP

Point-to-Point / Multipoint Interfaces

Like Frame Relay, ATM supports two types of interface: point-to-point and multipoint. The one you choose determines whether you need to use the configuration commands that ensure IP-to-ATM mappings. After configuring the PVC itself, you must tell the router which PVC to use in order to reach a specific destination. Consider these options:

1. Point-to-point subinterface - With point-to-point subinterfaces, each pair of routers has its own subnet. If you put the PVC on a point-to-point subinterface, the router assumes that there is only one point-to-point PVC configured on the subinterface. Therefore, any IP packets with a destination IP address in the same subnet are forwarded on this virtual circuit (VC). This is the simplest way to configure the mapping and is therefore the recommended method.
2. Multipoint networks - Multipoint networks have three or more routers in the same subnet. If you put the PVC in a point-to-multipoint subinterface or in the main interface (which is multipoint by default), you need to either configure a static mapping or enable inverse Address Resolution Protocol (ARP) for dynamic mapping.

Inverse ARP on ATM Connections

On Ethernet networks, IP-based network devices use ARP when they know the destination layer 3 address and need to discover the destination MAC address. Layer 2 network devices use inverse ARP (InARP) when they know the destination MAC address and need to discover the destination layer 3 address.
On ATM networks, RFC 1577 (Classical IP and ARP over ATM), specifies mechanisms for address resolution and defines the Inverse ATM Address Resolution Protocol (InATMARP).
With InATMARP, the ATM interface knows the layer 2 address. This is the PVC’s virtual path identifier (VPI) or virtual channel identifier (VCI). However, it still needs to discover which IP address is reachable at the remote end of a connection. To do this, the router sends an InATMARP request over a virtual connection for the address of the other end.
Note: InATMARP is the same protocol as Ethernet InARP. This is defined in RFC 1293, with additional extensions to support ARP in an ATM network.

Neither a static mapping nor InARP are required on a point-to-point subinterface since there is a single VC (sh atm vc) and a single path for the traffic. The router simply consults the routing table and makes a forwarding decision: 

sh atm map # with P2P ATM interface output of this command is empty

show run int a2/0.3 

interface ATM2/0.3 point-to-point

  ip address 192.168.3.1 255.255.255.252 

  no ip route-cache 

  no ip mroute-cache pvc 0/300 !

InARP is enabled on multipoint links by default.

show atm map 

 Map list ATM1/1/0.100_ATM_INARP : DYNAMIC 

  ip 1.1.1.2 maps to VC 19, VPI 2, VCI 100, ATM1/1/0.100 

 Map list ATM1/1/0.200_ATM_INARP : DYNAMIC 

  ip 2.2.2.2 maps to VC 20, VPI 2, VCI 200, ATM1/1/0.200

The show atm map command displays the dynamic mapping through InATMARP, while the show arp and show atm arp commands do not.

Static IP to ATM VC Mappings

Static map lists are a Cisco IOS Software feature that offers an alternative to using the ATMARP and InATMARP mechanisms. Using static maps, you can associate a protocol address with an ATM address on a switched virtual circuit (SVC), or with a VPI or VCI on a PVC.
Note: Static map lists do not relate to RFC 1483 or RFC 1577 .

interface ATM1/1/0.200 multipoint 

  ip address 2.2.2.1 255.255.255.0 

  no ip directed-broadcast 

  pvc 2/200 

    inarp 5 

    protocol ip 2.2.2.2 broadcast

You can check the mapping using the show atm map command. As you can see, the mapping of layer 3 to layer 2 addresses is permanent rather than dynamic, as it was when you used InARP.

show atm map 

 Map list ATM1/1/0.100_ATM_INARP : DYNAMIC
ip 1.1.1.2 maps to VC 19, VPI 2, VCI 100, ATM1/1/0.100
Map list ATM1/1/0.200pvc20 : PERMANENT
ip 2.2.2.2 maps to VC 20, VPI 2, VCI 200, ATM1/1/0.200, broadcast

Using Point-to-point subinterface with LLC/SNAP RFC1483

LLC/SNAP RFC1483 sometimes called MER (MAC Address Encapsulation) or Enet encapsulation in some modems. To setup Bridging in Cisco Router:

interface ATM0
 no ip address
 atm ilmi-keepalive
interface ATM0.1 point-to-point
 description ===WAN===
 ip address 172.16.16.16 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip verify unicast reverse-path
 atm route-bridged ip
 pvc 7/34
  encapsulation aal5snap
 !
end

To verify that router sees other end over ATM:
sh adjacency ATM0.1
Protocol Interface                 Address
IP       ATM0.1                    172.16.16.1(10)