Cluster 3. Network Switches.

You can use any switches that support VLAN and multicasting groups (also it's possible to use unicast at least with corosync 2.4.0). I'll use Cisco Catalyst 2960-S stackable switches (they form a stack - which can be managed as one switch).
We will use 2 Cisco stack switches (you can use just one switch) in order to make links redundant - eno4 of nodes must be connected to different switches, eno1, eno2, eno3 must be connected to the switch where the other node's eno4 interface is connected.

To setup stack:

1. Connect stack cables to the proper ports Stack1 to Stack1 and Stack2 to Stack2
2. Connect console cable to the Cisco and to your PC/NB
3. power on switches
4. verify that switch knows whe is a mater and who is a slave:
1. do sh sw
2. Master switch  Role must be - Master, current state - Ready
3. Slave switch Role must be - Member, current state  - Ready
4. Also verify that MSTR led on master switch is green
5. Verify stack-ports:
1. do sh sw stack-ports - all ports must be - OK
2. power-off master switch and verify that slave becomes master:
1. do sh sw
2. Removed switch Role must change to - Member, State must change to - Removed
3. Remaining switch Role must change to - Master, State must change to - Ready

Now you can setup stack from any switch:

1. name stack like agrp-stack01 (agrp is 4 letter owner code, stack01 is simply serial number of the stack, it's our first stack and because of that stack serial number is 01):
1. hostname agrp-stack01
2. configure username and password (simple passwords are giver for reference only):
1. aaa new-model
2. aaa authentication login default local
3. username admin privilege 15 secret 123456
4. enable secret 12345
3. create VLANs:
1. vlan 100
2. name BCN
3. vlan 200
4. name SN
5. vlan 51
6. name IFN
7. vlan 1000
8. name Deafult
4. disable all interfaces:
1. int range gi 1/0/1 - 28, gi 2/0/1-28
2. shut
3. do wr
5. enable needed interfaces and make them member of the needed VLAN, also disable STP on ports (STP requires blocking traffic to prevent loops, such a behavior can cause nodes think other node is dead while it's however alive):
1. disable STP for BCN and SN:
1. no spanning-tree vlan 100
2. no spanning-tree vlan 200
3. no spanning-tree vlan 51
2. port 1/0/24 & 2/0/24 will be IFN uplink ports - going to the other switch:
1. int ra gi 1/0/24 , gi 2/0/24
1. no shut
2. create LAG/bonding as in my case other switch is single switch, not a stack:
3. channel-group 1 mode on
2. configure Port-Channel1:
1.  int Po1
2. sw mode access
3. sw nonegotiate
4. sw access vlan 51
5. show etherchannel summary:
1. "Ports" must be - Gi1/0/24(P) Gi2/0/24(P) - meaning that both ports are bundled in a LAG
3. Configure iLO port - port 1/0/17 & 2/0/17 will be iLO interface connected ports (here we will insert node1 iLO to the switch1 and node2 iLO to the switch2):
1. node1 iLO - 1/0/17 - label this cable c01n01_ipmi
2. node2 iLO - 2/0/17 - label this cable c01n02_ipmi
3. int ra gi 1/0/17 , gi 2/0/17
1. no shut
2. sw nonegotiate
3. sw mode access
4. sw access vlan 100
4. int gi1/0/17
1. description agrp-c01n01
5. int gi2/0/17
1. description agrp-c01n02

Switch ports will be used as 5 ports per cluster, so that we can use each switch stack for serving up to 4 clusters:

• 1st cluster - gi1/0/1-4,17 & gi2/0/1-4,17
• 2nd cluster - gi1/0/5-8,19 & gi2/0/5-8,19
• 3rd cluster - gi1/0/9-12,21 & gi2/0/9-12,21
• 4th cluster - gi1/0/13-16,23 & gi2/0/13-16,23

We have two options to further setup our stack, one is for Linux bonding and bridging and the other  is for Open vSwitch.

Linux bonding and bridging

Subnet	VID	NIC	Link 1	NIC	Link 2	Bond	Net IP
BCN	100	eno1	bcn_link1	eno4	back_link.100	bcn_bond	10.10.53.0/24
SN	200	eno2	sn_link1	eno4	back_link.200	sn_bond	10.10.52.0/24
IFN	51	eno3	ifn_link1	eno4	back_link.51	ifn_bond	172.16.51.0/24

1. port 1/0/1 & 2/0/1 will be BCN ports:
1. node1 eno1 - 1/0/1
2. node2 eno1 - 2/0/1
3. int ra gi 1/0/1 , gi 2/0/1
1. no shut
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 100
5. sw trunk native vl 1000
2. port 1/0/2 & 2/0/2 will be SN ports:
1. node1 eno2 - 1/0/2
2. node2 eno2 - 2/0/2
3. int ra gi 1/0/2 , gi 2/0/2
1. no shut
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 200
5. sw trunk native vl 1000
3. port 1/0/3 & 2/0/3 will be IFN ports:
1. node1 eno3 - 1/0/3
2. node2 eno3 - 2/0/3
3. int ra gi 1/0/3 , gi 2/0/3
1. no shut
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 51
5. sw trunk native vl 1000
4. port 1/0/4 & 2/0/4 will be backup ports (here we will insert node1 port eno4 to the switch2 and node2 eno4 to the switch1):
1. node1 eno4 - 2/0/4
2. node2 eno4 - 1/0/4
3. int ra gi 1/0/4 , gi 2/0/4
1. no shut
2. sw nonegotiate
3. sw mode trunk
4. sw trunk allowed vlan 100,200,51
5. sw trunk native vl 1000

Open vSwitch

We will bond all 4 interfaces (from eno1 through eno4) to the OvS bond - ovs_bond.
And then we'll create OvS internal ports and assign them IP:

Subnet	VID	OvS internal port	Net IP
BCN	100	bcn-bond1	10.10.53.0/24
SN	200	sn-bond1	10.10.52.0/24
IFN	51	ifn-bond1	172.16.51.0/24

1. ports Gi1/0/1-1/0/4 and Gi2/0/1-2/0/4 will be trunk ports carrying all VLANs:
1. node1 eno1 and eno3 - 1/0/1 & 1/0/3 - label this 2 cables eno1_c01n01_ovs_bond and eno3_c01n01_ovs_bond
2. node1 eno2 and eno4 - 2/0/2 & 2/0/4 - label this 2 cables eno2_c01n01_ovs_bond and eno4_c01n01_ovs_bond
3. node2 eno1 and eno3 - 2/0/1 & 2/0/3 - label this 2 cables eno1_c01n02_ovs_bond and eno3_c01n02_ovs_bond
4. node2 eno2 and eno4 - 1/0/2 & 1/0/4 - label this 2 cables eno2_c01n02_ovs_bond and eno4_c01n02_ovs_bond
5. int ra gi 1/0/1, gi 1/0/3, gi 2/0/2, gi 2/0/4
1. description agrp-c01n01
2. channel-group 2 mode active #enabling LACP use different channel-group numbers for nodes
3. no shut
6. int Po2
1. description agrp-c01n01
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 100,200,51
5. sw trunk native vl 1000
6. no shut
7. int ra gi 1/0/2, gi 1/0/4, gi 2/0/1, gi 2/0/3
1. description agrp-c01n02
2. channel-group 3 mode active #enabling LACP use different channel-group numbers for nodes
3. no shut
8. int Po3
1. description agrp-c01n02
2. sw mode trunk
3. sw nonegotiate
4. sw trunk allowed vlan 100,200,51
5. sw trunk native vl 1000
6. no shut
9. sh int port-channel {1|2|3} # to view info about LAG interfaces (choose needed LAG number)

LACP bandwidth - the maximum through-output will remain equal to the through-output of the single link. In fact you get more lanes to move but the maximum speed remains the same. By enabling LACP you increase maximum overall bandwidth. This achieved using load-balancing (Cisco default LB mechanism is source-MAC balancing).

This tutorial was used to understand and setup clustering: AN!Cluster