RAFT algorithm/protocol

 Based on notes made upon: http://thesecretlivesofdata.com/raft/

Consensus - agreement on a value:

1. non-distributed - DB client send data to single DB server
2. distributed - DB client send data to single DB server and this server must replicate data to the other DB servers

In Raft protocol node can be in 3 states:

1. follower
2. candidate
3. leader

Leader Election Process:

1. Each node begins in follower state and starts election timeout (random between 150ms-300ms)
2. If the follower doesn't hear from leader (during election timeout), it becomes candidate and starts new election term
3. The candidate votes for itself and sends Request-Vote message to the other nodes and starts election timeout
4. Other nodes reply with their vote if they didn't vote already in this term
5. If:
1. The candidate gets votes from the majority of nodes - it becomes leader and stops timeout
2. Election term is not properly held (no leader is elected) during election timeout (possibly with even number of nodes, when two nodes become candidate at the same time and can't collect majority of votes) - new election term is started because of election timeout
6. Majority of votes forces only one leader to be selected per term
7. All changes to the system now goes through the leader and heartbeat timeout is started on followers
8. Election term lasts until leader manages to send Append-Entry message during heartbeat timeout, if so - followers restart heartbeat timeout, otherwise new election term starts
9. If node goes down and then resumes, it firstly checks it's own election term number with the number of the election term in the Append-Entry message. If term number is equal or higher than it's own term number - current leader is accepted and all values are updated according to the new leader's values

Log Replication:

1. Each change is added as an uncommitted entry (value is not updated) to the leader's log
2. Leader sends replication to the followers
3. Leader waits until the majority of followers made uncommitted entry to their own log
4. Leader commits entry and value changes
5. Leader sends Append-Entry messages to the followers during heartbeat timeout (Append-Entry messages sent to the followers even no changes are really made - in order to keep-alive)
6. Followers commit entry and respond to each Append-Entry message

Linux file/directory permissions and extended permissions/ACL (getfacl/setfacl)

 [admino@mwg www]$ getfacl SystemFiles/

# file: SystemFiles/

# owner: www-data

# group: www-data

user::rwx

group::rwx

other::r-x

#Give access to admino user:

#-m is modify ACL (add or change)

#-x is delete ACL (setfacl -x u:sales:r test/)

#-d is to put a 'default' user and/or group to new generated files (can be given only to directory).

# u: / g: / o: / m: / d: are user, group, others, effective mask, default (the same as -d)

#

#read (r or 4) – read file; list directory content

#write (w or 2) – modify file; if x is also set - modify dir contents (rm, cp, mv files and subdirectories), otherwise - no effect

#execute (x or 1) – execute file, enter directory

#The setuid (s instead of u:x) bit simply indicates that when running the executable, it will set its permissions to that of the user who created it (owner), instead of setting it to the user who launched it. Similarly, there is a setgid (s instead of g:x) bit which does the same for the gid. (-rwsr-xr-x. 1 root root 27856 Apr  1  2020 /usr/bin/passwd).

#When a directory has the sticky (t instead of o:x) bit set, its files can be deleted or renamed only by the file owner, directory owner and the root user (drwxrwxrwt.   6 root root  4096 Jul  1 09:59 tmp). 

#ls shows + sign after permissions to indicate extended ACL (like: drwxr-xr-x+) s

###Remove all existing extended ACL:

sudo setfacl -b /var/www/SystemFiles

###Get list of extended and standard ACL:

getfacl /var/www/SystemFiles

getfacl: Removing leading '/' from absolute path names

# file: var/www/SystemFiles

# owner: www-data

# group: www-data

user::rwx

group::rwx

other::r-x

###give admino ability to upload files:

sudo setfacl -m u:admino:rwx /var/www/SystemFiles

###

getfacl: Removing leading '/' from absolute path names

# file: var/www/SystemFiles

# owner: www-data

# group: www-data

user::rwx

user:admino:rwx

group::rwx

mask::rwx

###to read non-default ACL:

getfacl SystemFiles | grep -Ev "(::)|^#|^$"

###set defaults for user and group:

sudo setfacl -m d:u:www-data:rw SystemFiles

sudo setfacl -m d:g:www-data:rw SystemFiles

###check ACL (effective means group::rwx applies mask::rw and we have union - effective:rw):

[admino@mwg www]$ getfacl SystemFiles/u34jf\ o.png

# file: SystemFiles/math.png

# owner: admino

# group: admino

user::rw-

user:www-data:rw-

group::rwx                      #effective:rw-

group:www-data:rw-

mask::rw-

other::r--

Asterisk ARI Introduction

 To enable:

Asterisk HTTP server is used to access ARI, WS, AMI. If you need WebSocket: The built-in Asterisk HTTP server is used to provide the WebSocket support. The res_http_websocket must also be built and loaded by Asterisk. For most individuals this is done by default.

You can’t view WebSockets in a standard browser. That's how the spec is written. If you view it in a standard browser the server will tell the protocol it needs to upgrade. This is to keep it in line with the HTTP spec. It doesn’t mean there is an upgrade. So don't try to access ws://yourAsteriskServerIP:8088/ws from a standard browser.

OpenAPI - specification standardizing describing of the REST API-services. This describtions can be in JSON or YAML.

Swagger – framework allowing you to describe the structure of your APIs using OpenAPI specification so that machines can read them. Also structure become more readable by humans via Swagger UI.

Don’t access ARI directly from a web page It’s very convenient to use ARI directly from a web page for development, such as using Swagger-UI, or even abusing the WebSocket echo demo to get at the ARI WebSocket. But, please, do not do this in your production applications. This would be akin to accessing your database directly from a web page. You need to hide Asterisk behind your own application server, where you can handle security, logging, multi-tenancy and other concerns that really don’t belong in a communications engine.

http.conf
[general]
enabled=yes
enablestatic=no
bindaddr=yourAsteriskServerIP
bindport=8088
servername=Asterisk 

ari.conf

[general]

enabled = yes

pretty = yes

allowed_origins = yourDeveloperHost

[testARI]

type = user

read_only = no

password = Sup3rM3ga$tr0ng

password_format = plain

asterisk -rx "module reload res_ari.so"

asterisk -rx "reload http"

It's highly recommended to use TLS and secured web socket (WSS)  server in the production environment:

openssl req -new -x509 -days 365 -nodes -out /tmp/foo.pem -keyout /tmp/foo.pem

mv /tmp/foo.pem  /var/lib/asterisk/

http.conf

tlsenable=yes

tlsbindaddr=yourAsteriskServerIP:8089

tlscertfile = /var/lib/asterisk/foo.pem

tlsprivatekey = /var/lib/asterisk/foo.pem

asterisk -rx "reload http"

asterisk -rx "http show status"

To check:

asterisk -rx "http show status"

asterisk -rx "ari show status"

From browser:

yourServerIP:8088/ari/api-docs/resources.json

To read in more user-friendly way:

1. in ari.conf add http://ari.asterisk.org to the allowed_origins (using coma as origin separator).
2. asterisk -rx "module reload res_ari.so" 
3. access http://ari.asterisk.org enter yourAsteriskServerIP:8088/ari/api-docs/resources.json into the first input-placeholder, then username:password of the ARI to the second input-placeholder
4. press "Explore"

Stasis Message Bus:

In Asterisk 12, a new core component was added to Asterisk: the Stasis Message Bus (SMB). As the name suggests, Stasis is an internal publish/subscribe message bus that lets the real-time core of Asterisk inform other modules or components – who subscribe for specific information topic – about events that occurred that they were interested in (for example: Stasis Message: Channel Hangup). SMB is used by AMI, CDR, CEL, ARI etc. Key concepts of the SMB:

1. Message - group name of the event (ax: Channel Hangup)
2. Publisher - source of the event (ex: Channel Core - publishes message)
3. Subscriber - subscribes to a topic and receives messages of this topic (ex: AMI subscribed to channel event topic)
4. Cache - Some Messages - particularly those that affect core communications primitives in Asterisk (such as channels or bridges) are stored in a special cache in Stasis. Subscribers have the option to query the cache for the last known state of those primitives.

Using Stasis diaplan application:

extensions.conf

[ARI-test]

exten => 1000,1,NoOp() 

same => n,Answer() 

same => n,Stasis(hello) 

same => n,Hangup()

 asterisk -rx "dialplan reload"

Some notes:

1. You can't manage channel which is outside of the Stasis app
2. You can't retrieve data from channels which are outside of the Stasis app
3. When dialplan is in Stasis app - dialplan stays in until you exit Stasis
4. Communication between app inside Stasis and client is done over WebSocket in asynchronous manner
5. If nobody is connected to the app inside Stasis ("hello" app in the dialplan above) over WebSocket  - app will stop working and handover execution to the dialplan

If we have an endpoint in ARI-test context and make call to the 1000, we'll see the following in Asterisk CLI:

ERROR[39437][C-00000003]: res_stasis.c:1325 stasis_app_exec: Stasis app 'hello' not registered

The Stasis application is how a channel goes from the dialplan to a Stasis application. When a channel enters the Stasis application in the dialplan, a StasisStart event is sent to the application's associated WebSocket. So as written above - we must to have our own application which will connect to the WS serer and interact with ARI.

ARI Resources:

Resources below must be controlled by ARI in order to be reachable:

1. asterisk - global variables,server info, configuration files etc.
2. endpoints - list endpoint, send message etc.
3. channels - list, create, redirect, answer, mute, ring, hold, moh, get variable, dial etc.
4. bridges - construct sharing media among channels: list, create, add channel, remove channel etc.
5. recordings - list, copy, stop, pause etc.
6. sounds - list, get details etc.
7. playbacks - playback control
8. deviceStates - list, change etc.
9. mailboxes - list, destroy etc.
10. events - WbSocket events: generate etc.
11. applications - list, subscribe etc.

Python aioari / ari:

With Python3:

pip install aioari # this is the latest version of ari.py using async swagger.py

Wiht Python2:

pip install ari

import aioari

import asyncio

client = await aioari.connect('http://yourServerIP:8088', 'username', 'password')

def stasis_start(obj, event):

    print('start')

def stasis_end(obj, event):

    print('stop')

client.on_channel_event('StasisStart', stasis_start)

client.on_channel_event('StasisEnd', stasis_end)

loop = asyncio.get_event_loop() 

loop.run_until_complete(client.run(apps="hello"))

To close loop:

loop.close()

After entering above in python shell:

 Creating Stasis app 'hello'

  == WebSocket connection from 'hostAccessingARI:65292' for protocol '' accepted using version '13'

Then we call 1000 to enter Stasis hello app:

    -- Executing [1000@ARI-test:1] NoOp("PJSIP/2222-00000017", "") in new stack

    -- Executing [1000@ARI-test:2] Answer("PJSIP/2222-00000017", "") in new stack

    -- Executing [1000@ARI-test:3] Stasis("PJSIP/2222-00000017", "hello") in new stack

In ipython shell we'll see: start

Hangup call:

stop # in ipython shell

asterisk -rx "stasis show topic ari:application/hello"

Name: ari:application/hello

Detail:

Subscribers count: 1

Forwarding topic count: 0

Duration time: 00:03:58

Ctrl+C in ipython shell (don't forget to close loop):

 Deactivating Stasis app 'hello'

 Shutting down application 'hello'

  == WebSocket connection from 'hostAccessingARI:65292' forcefully closed due to fatal write error

 Destroying Stasis app hello

    -- Remove stasis-hello/h/1, registrar=res_stasis; con=stasis-hello(0x7f0dac014d70); con->root=0x7f0dac015160

    -- Remove stasis-hello/_./1, registrar=res_stasis; con=stasis-hello(0x7f0dac014d70); con->root=0x7f0dac015160

Hangup() will only be executed if we call 1000 without previously connecting to the WebSocket. Also below will be in Asterisk CLI:

ERROR[24668][C-00000019]: res_stasis.c:1325 stasis_app_exec: Stasis app 'hello' not registered

To extend functionality of ARI app, you can use this link:

GitHub - asterisk/ari-examples: ARI examples in Python and JavaScript.

Cisco DNA quick overview

SDN (Software Defined Networking) - attempts to centralize network intelligence in one network component (SDN Controller) by disassociating the forwarding process of network packets (data plane) from the routing process (control plane).

REST (Representational state transfer) is a software architectural style that defines a set of constraints to be used for creating Web services.

API (Application Programming Interface)

YANG (Yet Another Next Generation) is a data modeling language for the definition of data sent over network management protocols and it uses either XML or JSON encoding.

Cisco DNA (Digital Network Architecture) - is SDN for campus networks. DNA supports management through REST API using YANG model.

DNA Center (DNAC) is centralized network management system aimed to be simple and intuitive in order to be used for management of all network functions and optimization of network and applications. With DNA Cisco goes beyon traditional SDN and realizes Intent-Based Network.

SDA (Software Defined Access) - policies are applied to users and applications which makes management easier. SDA used for adding policies concerned with security, segmentation, access etc.

Cisco series used in SDA are:

1. Access - Catalyst 9200, 9300, 9400
2. Aggregation - Catalyst 9400, 9500, 9600
3. Core - Catalyst 9500, 9600, 9800

NDP (Network Data Platform and Assurance) - analytic platform responsible for data collecting. Collects and classifies huge amount of data sent over network (application, user and equipment data). Using data provided by NDP DNA Center Assurance produces analytics and operative information about network state and also makes forecasts. 

How typical network tasks are made with DNA:

1. Connecting network devices - device role is defined within DNAC or device can be bought with preinstalled Plug&Play agents which are needed for this particular device
2. Network devices software versioning - DNAC saves "gold" versions of software and administrator just applies update policies needed for particular devices or segments
3. Scale-able access policies - traditional access policies are IP or VLAN based. Big IP addresses pools add additional complexity. DNAC applies policies to the entire network - when some devices are changed - policies remain the same
4. Campus network segmentation - DNAC realizes that using access matrix for all device categories used in the network. When new device is added to the network segmentation is automatically made after adding this device to the needed group

MySQL DB-size, tables size

Size of all tables in particular DB (change NEEDED_DB_NAME to needed):

SELECT table_name AS "Table", 

data_length+index_length AS "Size in bytes" 

FROM information_schema.TABLES 

WHERE table_schema = "NEEDED_DB_NAME" 

ORDER BY (data_length + index_length) DESC;

Size of all DB in MySQL:

SELECT table_schema AS "Database", 

ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS "Size (MB)" 

FROM information_schema.TABLES 

GROUP BY table_schema;

Count of rows of all tables in particular DB ((change NEEDED_DB_NAME to needed):

SELECT table_name AS "Table", 

table_rows AS "Rows" 

FROM information_schema.TABLES 

WHERE table_schema = "NEEDED_DB_NAME" 

ORDER BY table_rows DESC;

Table size description (depends on system block size):

1. index_length=data_length=16384 (total size is 32768) - empty table with index
2. index_length=0, data_length=16384 - empty table without index (only primary key can exist)

In MySQL (INNODB):

KEY=INDEX

PRIMARY KEY=special case of UNIQUE KEY (identifies that row and also data can't be NULL)

To check index, engine, index_length, data_length, create_time (change NEEDED_TABLE_NAME):

show table status like 'NEEDED_TABLE_NAME';

DB Normal Forms (primitive description)

 Normal Forms (NF) helps to resolve possible problems with insert, update and delete DB operations. 

To be in the next level NF DB must already be in all previous levels of NF (to be in 3NF DB must already be in 1NF and 2NF).

There are plenty of normal forms but in most practical situations first three forms are enough to prevent possible problems.

1NF (first normal form)

1NFexample1: You have "children" table with "child" column storing something like "Child'sNameSurname"

1NFexample2: You have "children" table with "parent" column storing something like "Mom'sNameSurname, Dad'sNameSurname"

1NFexample3: You have "children" table with "parent1_name" and "parent2_name" columns storing something like "Mom'sName" and "Dad'sName"

To be in 1NF:

1. The relation has a Primary Key (PK), which uniquely identifies each row in the relation.
1. As in 1NFexample1 "child" may not be unique so you need to add "row_id" as PK
2. Every column stores Atomic Values
1. As in 1NFexample2 non-atomic values are in "parent" column, so we need to split this into to rows, each with "parent" column containing name of one parent
2. Also "parent" column must be separated into "parent_name" and "parent_surname" columns
3. There are no Repeating Groups.
1. As in 1NFexample3 "parent1_name" and "parent_name" are repeating groups, so we need to add "parent_name" column and for each parent add one row

2NF (first normal form)

2NFexample1: You have "children" table with CPK "child_name"+"zip" and also having "city" column

To be in 2NF:

1. DB must already be in 1NF
2. No partial dependencies (also called partial functional dependencies) are in table
1. That is, each non-key attribute is functionally dependent on the full PK (PK for 2NF can be only multi column PK - also called composite PK. If table is in 1NF and having single-column PK, then this table is also automatically in 2NF):
1. As in 2NFexample2 "city" is related to "zip" (part of CPK), but not related to the "child_name", so we must add new table "locations" with "location_id", "zip", "city" columns. And use "location_id instead of "zip" as part of CPK

3NF (first normal form)

3NFexample1: You have "children" table with "row_id" PK column and "city" and "zip" columns storing something like "Child'sCity" and "Child'sZip"

To be in 3NF:

1. DB must already be in 2NF
2. No transitive functional dependencies - non-key column depends (functionally) on another non-key column, which is (functionally) dependent on the PK
1. As in 3NFexample1 "zip" and "city" are related to "row_id" (PK), but also depend on each other, so we must add new table "locations" with "location_id", "zip", "city" columns. After that in "children" table we'll use "location_id" instead of "city" and "zip"

Short note about MySQL INDEX, KEY

MySQL KEY = INDEX - helps to find data faster via using indexing in needed columns

PRIMARY KEY (PRI) - repetitions in this column (or group of columns) are not allowed, NULL values are not allowed

UNIQUE KEY (UNI - one column) - repetitions in this column are not allowed, NULL values are allowed

UNIQUE KEY (MUL - multiple columns) - repetitions allowed in one column bit not in both at he same time, NULL values are allowed

INDEX=KEY - repetitions allowed, NULL values allowed

FOREIGN KEY = only value existing in other table can be added to this column