Linux tar incremental backup & restore

1. Backup

In this post I'll describe steps to use tar for incremental backup of files.

Incremental backups needed when it's lack of free space on your backup device so you have full-backup and incremental backup which contains changes/differences.

To make incremental backups tar uses `.snar` meta-files which contains changes since last backup. We'll use following tar options for backup (-vcSpzg):

1. -c OR --create - tells that we are creating new archive.
2. -v | --verbose - useful if you want log files.
3. -S OR --sparse - sparse files are files having holes (the section of the files' contents which was never written, the contents of a hole reads as all zeroes). Many file systems not actually allocating storage for hole but saving length of the hole so without that option tar archive will be longer than the original. Using this option makes tar to recognize sparse files and save only actual date. When restoring tar will add holes to such files at the needed places.
4. -p OR --preserve-permissions - we tell tar to save all permissions for every file.
5. -z OR --gzip - use compression.
6. -g OR --listed-incremental - show which meta-data file to use, this options show that we make backup and show's it's level. 
7. -f OR --file - backup file name.

The core concept is:

1. if snar file in `-g` option is not existing - make full backup and create snar file containing list of files backuped
2. if snar file exists - make incremental backup not backuping files listed in snar file

We'll have 7 files:

1.  for Monday - dir-dump1.sh
2.  for Tuesday - dir-dump2.sh
3.  for Wednesday - dir-dump3.sh
4.  for Thursday - dir-dump4.sh
5.  for Friday - dir-dump5.sh
6.  for Saturday - dir-dump6.sh
7.  for Sunday - dir-dump7.sh

So for example if we want to make full backup on Monday:
cat dir-dump1.sh
#! /bin/bash
#remember core concept (1) - we removing snar file for full-backup day
/bin/rm -f /mnt/backup/snar-dir/monday0.snar

#monday0.snar backup file will be created during full-backup
#$(date +%W) returns number of the week of the year
/bin/tar -cvSpzg /mnt/backup/snar-dir/monday0.snar -f /mnt/backup/$(date +%W)-monday.gz   /var/lib/recordings

For Tuesday we make incremental backup with file differing from monday backup:
cat dir-dump2.sh
#! /bin/bash
#core concept (2) - specified snar file will be used for incremental backup
/bin/cp -f /mnt/backup/snar-dir/monday0.snar /mnt/backup/snar-dir/tuesday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/tuesday.snar -f /mnt/backup/$(date +%W)-tuesday.gz   /var/lib/recordings


For Wednesday, Thursday, Friday, Saturday & Sunday we make the same - renaming snar file of the previous day:

cat dir-dump3.sh
#! /bin/bash
/bin/mv -f /mnt/backup/snar-dir/tuesday.snar /mnt/backup/snar-dir/wednesday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/wednesday.snar -f /mnt/backup/$(date +%W)-wednesday.gz   /var/lib/recordings

cat dir-dump4.sh
#! /bin/bash
/bin/cp -f /mnt/backup/snar-dir/wednesday.snar /mnt/backup/snar-dir/thursday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/thursday.snar -f /mnt/backup/$(date +%W)-thursday.gz   /var/lib/recordings

cat dir-dump5.sh
#! /bin/bash
/bin/cp -f /mnt/backup/snar-dir/thursday.snar /mnt/backup/snar-dir/friday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/friday.snar -f /mnt/backup/$(date +%W)-friday.gz   /var/lib/recordings

cat dir-dump6.sh
#! /bin/bash
/bin/cp -f /mnt/backup/snar-dir/friday.snar /mnt/backup/snar-dir/saturday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/saturday.snar -f /mnt/backup/$(date +%W)-saturday.gz   /var/lib/recordings

cat dir-dump7.sh
#! /bin/bash
/bin/cp -f /mnt/backup/snar-dir/saturday.snar /mnt/backup/snar-dir/sunday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/sunday.snar -f /mnt/backup/$(date +%W)-sunday.gz /var/lib/recordings

Then add backup for every day to the crontab (redirecting all errors and output to the log file):
0 5 * * 1 /mnt/backup/dir-dump1.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 2 /mnt/backup/dir-dump2.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 3 /mnt/backup/dir-dump3.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 4 /mnt/backup/dir-dump4.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 5 /mnt/backup/dir-dump5.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 6 /mnt/backup/dir-dump6.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 7 /mnt/backup/dir-dump7.sh 2>&1>> /mnt/backup/dir-dump.log

If you understood core concepts you can do any mix of full and incremental backups, using snar files.

 

2. Restore

Meta-files not needed for restoring. They needed only for the incremental backup process.  To restore we first restore last full backup and then all incremental backups. If use our backup scheme as example, restore will be as (i.e. we restore files for 10th week of the year). We'll ise `/` in `--directory` option because tar saves full-path and showing `/var/lib/recordings` will result in having `/var/lib/recordings/var/lib/recordings/file_name`. For snar file we specify `/dev/null` it is necessary because option `-g` must be specified in order to keep-track on operations over files during incremental backup (deletion, creation, changing) and snar file option can't be empty if you specify `-g` option:

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-monday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-tuesday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-wednesday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-thursday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-friday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-saturday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-sunday.gz  --directory=/

 

Time-Management & Productivity 1

Last month I realised how much time we loose doing things we don't need to do, trying to procrastinate (do anything but task we need to accomplish).  Many times I tried to begin using to-do lists, many different approaches  like Kanban, GTD, Eisenhower Box, The Pomodoro Technique etc.  But all this approaches failed for me (to speak frankly may be I'm too lazy and trying to find bad sides in everything). So I decided to find/set-up my own approach, based on other approaches learnt by me. I'll try to write new posts every time I used all things mentioned in the previous posts.

Lets start...
First of all you need to write-down (I'll use the Google Docs spreadsheets - for easiness of synchronisation across devices) all things you are planning to do. Write everything - you just want to empty your brain (as David Allen's GTD methodology suggests). Write even if it's not actually a todo-item, it may be just your thought on something - to the `backlog` (Kanban term) column.

Next make processing off items in this column (using Eisenhower Box) and make note in the `type` column (select B column then right click > Data validation...

• Cell range: > Add: 
• Sheet1!B2:B 
• Criteria > List of Items > Add:
• DoFirst,Schedule,Delegate,MayBeSomeDay,Done)

DoFirst (important and urgent) - must be done in day or two
Schedule (important but not urgent - things that you really need to do)  - write deadline in `note` column
Delegate (urgent but not important) - delegate this if possible (write name of person in `note` column)
MayBeSomeDay (not important and not urgent - at least for now) - if you can just delete this item from the backlog or it may be plan for future spare-time (i.e. make ASA config more readable - good but not too important or urgent to put it into schedule).
Done - mark all done items

Daily review your list and try to do all items with DoFirst and Schedule type, also check Delegate items. View MayBeSomeDay items when you have spare time.

PS I'll try to do things in this post for a period of one month and then do changes to this work-flow.

Update 1:
decided to use range for `Data validation...`
Created sheet `REF` with `DoFirst,Schedule,Delegate,MayBeSomeDay,Done` in an A column (each item in it's own cell).
Then removed data validation, and:
select B column then right click > Data validation...

• Cell range: > Add: 
• Sheet1!B2:B 
• Criteria > List from a range > Add:
• REF!A:A

Update 2:
have no more than 5 items in `DoFirst`, if you have more, then prioritise this items and move less important to the `Schedule`. Try to have as less items in `Schedule` as possible (move them to `MayBeSomeDay` if you can). When you full-fill `DoFirst` you can review `Schedule` items and choose most important to be new `DoFirst`. 

Update 3:
Added `DailyRoutine` to the `REF!A:A` for now I'll have only one routine - daily review of the items in the list.

Update 4:
Don't do daily review if you didn't do all `DoFirst` items. For now I have 3 `DailyRoutine`s (daily review, Amosov exercies and CCNA 100-105 cert prepare) and 1 `DoFirst` and 1 `WeeklyRoutine` (Choose DoFirst for weekedns and holidays and then take a photo of this. I do this because don't want to spend additional time on day-offs) item. This all helps me to avoid distraction

Update 5:
Added `PreSelected` to the `REF` using it for quickly selecting `DoFirst`s (first mark as `PreSelected` and then after doing current `DoFirst` selecting new one from `PreSelected`). Using conditional formatting to make easy understanding done and awaiting routines.

Cisco IOS (SLA / Track / default route change). Using alternative default route on fly.

Many branches were configured only with default route  of the `master` ASA and no other routes (so-called stub network). When migrating from old ASA to the new one I needed to make this migration `painless` (minimal downtime).

These are steps to change default routes on the routers:

#configure SLA:
#ping default-gateway

ip sla 1
 icmp-echo 172.16.0.8

exit

ip sla schedule 1 life forever start-time now

#configure tracker of this SLA:

track 1 ip sla 1 reachability

#configure main route with track number 1 :

ip route 0.0.0.0 0.0.0.0 172.16.0.8 track 1

#delete old default route:

no ip route 0.0.0.0 0.0.0.0 172.16.0.8

#add backup route with higher metric:

ip route 0.0.0.0 0.0.0.0 172.16.0.8 10

When  main route is unreachable, backup route with metric 10 will be used.

After finishing migration we can completely remove old route and add new route with default metric:

ip route 0.0.0.0 0.0.0.0 172.16.0.8

after issuing this command route with metric 10 is automatically removed.

Then on an old ASA deny all needed protocols for migrating router. In my environment:

access-list regions.acl line 1 extended deny esp host <needed-IP> any

access-list regions.acl line 1 extended deny ip host <needed-IP> any

access-list regions.acl line 1 extended deny icmp host <needed-IP> any

It's obvious that this is only one step of the migration process but other steps were more obvious for me than  this one.

Cisco IOS (SLA / Track / event manager). Restarting Interface on fail.

Recently it appeared that one of the branches I'm responsible for loses link on one of interfaces. I don't know why but when interface is `shutdown-noshutdown`-ed everything becomes OK. So I looked for a way to `restart` interface automatically:

#First we'll create SLA (Service Level Agreement)
#sla number is `1`
#it will ping `172.16.0.70` from `172.16.0.77` IP source address
#ping will be issued every 3sec
#threshold must be < timeout
#threshold is RTT (Round Trip Time - time needed to reach the other side and to return) in msec - needed for SLA Statistics and Monitoring
#timeout - if no answer in 1.5sec - consider it as failed attempt
ip sla 1
 icmp-echo 172.16.0.70 source-ip 172.16.0.77
 threshold 1000
 timeout 1500
 frequency 3
exit
#Then we'll make it endless and start it right now:
ip sla schedule 1 life forever start-time now
#show commands fro SLA:
sh ip sla configuration  1
sh ip sla statistics 1

#Now we'll configure tracker of the SLA
# track number 10
#uses SLA number 1
#consider down if down for 10sec
#consider up if up for 60sec
track 10 ip sla 1 reachability
 delay down 10 up 60
#to view:
sh track brief
sh track 10

#Configure event manager (response on event)
#event manager named restart_interface
#will use track number 10
#restart_interface will be executed if track 10 state is down
event manager applet restart_interface
 event track 10 state down
 action 0   cli command "enable"
 action 1   cli command "conf t"
 action 2   cli command "int fa0/1"
 action 2.5 cli command "shut"
 action 3   cli command "no shut"
 action 4   cli command "end"
#to view:
sh even manager policy registered
sh event manager history events

MySQL files extensions

grep data  /etc/my.cnf
datadir=/var/lib/mysql

We can finf below files in the `/var/lib/mysql/<db_name>`:

db.opt
this file contains characteristics of database like so:
default-character-set=latin1
default-collation=latin1_swedish_ci
every MyISAM table will have frm / MYD /  MYI and TRG (if this table has trigger)
.frm :
MySQL represents each table by an .frm table format file, stores table definition in the database directory. It is stored as  .frm under data directory.

.MYD :
This is the extension of the Data files for MyISAM tables.

.MYI :
This is the extension of the Index files for MyISAM tables.

<table_name>.TRG
contains SQL script of trigger for this table

<trigger_name>.TRN
contains a table name this trigger belongs to

.ibd :
If you specify innodb_file_per_table option to my.cnf, InnoDB stores each table in its own .ibd file in the same MySQL database directory where the .frm file is created. For InnoDB tables, the data and the indexes are stored together in the .ibd file.
 

MySQL root password recovery

You can recover MySQL database server root password following below six easy steps:

1. Stop the MySQL server process:
1. /etc/init.d/mysqld stop
2.  Start the MySQL (mysqld) server/daemon process with the --skip-grant-tables option so that it will not prompt for password:
1.  mysqld_safe --skip-grant-tables &
2.  /etc/init.d/mysqld start
3.  Connect to mysql server as the root user:
1.  mysql -u root
4.  Setup new mysql root account password i.e. reset mysql password:
1. mysql> use mysql;
2. Change root password (read all 3 below before issuing commands):
1. MySQL 5.7.6 and later: ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewPass';
2. MySQL 5.7.5 and earlier: SET PASSWORD FOR 'root'@'localhost' = PASSWORD('MyNewPass'); 
3. Or for any MySQL version: update user set Password=PASSWORD('MyNewPass') where User='root';
3. mysql> flush privileges;
4. mysql> quit 
5. Exit and restart the MySQL server:
1.  /etc/init.d/mysqld restart
6.  Connect to MySQL and verify if new root password is accepted:
1. mysql -u root -p

MySQL Slow connection

When connection to MySQL server is too slow it may be because of name resolving (at least attempt to resolve names). To change this behaviour:

edit my.cnf:

Add this line under the `[mysqld]` section:

skip-name-resolve

And then restart mysqld daemon

Cisco ASA setting interfaces up and understanding security-level and access rules

Simple steps to set-up Cisco ASA interfaces:

interface GigabitEthernet0/0
 nameif link_to_asa
 security-level 100
 ip address 172.28.0.1 255.255.255.0
 no shutdown

interface GigabitEthernet0/1
 nameif link_to_region
 security-level 90
 ip address 172.16.43.26 255.255.255.0
 no shutdown

nameif shows alias for the interface, which can be used in access-group, access-list, nat commands.

security-level from HIGH level to the LOW level traffic and return traffic is permitted (no access-list needed but you can add if you want and acl is checked before the security-level), from LOW level to the HIGH level traffic is dropped. Return traffic is permitted for bidirectional connections (TCP / UDP - connections are tracked).
 In routed mode (ASA has routed and transparent mode) security-levels are implicit permitting or denying only IPv4 and IPv6 traffic (in transparent mode ARP, STP traffic controlled by security-levels). To control other types of traffic you need to set-up access-lists.

In Cisco ASA ICMP is treated as unidirectional connection so you need to permit ICMP in both directions. To allow ICMP to ASA interfaces itself use:
`icmp permit <IP or Net_IP/mask> <ICMP destination ASA interface name>`
i.e. : 

icmp permit 172.16.43.0 255.255.255.0 link_to_asa

By default ICMP ping is permitted to all ASA interfaces

Access-lists
Access-list can be per interface (must specify direction when applying -> `in` or `out`) and global.
Global acls based on source and destination IP addresses and not using interface names and disabling all security-level affects because of implicit deny at the end. But you still need `same-security-traffic permit inter-interface` to permit communication between different interfaces with the same security level and `same-security-traffic permit intra-interface` to permit communication between peers connected to the same interface.
If you'll apply both per-interface and global access-lists, then arriving packet first goes through interface acl, then through global acl and in the end to the implicit deny of the global acl (interface acl has no implicit deny rule).
Access-lists don't have implicit deny and we must take security-levels into consideration:
Traffic from the higher to the lower will be dropped only if ACL on this lower security-level interface denies traffic from higher security-level interface and vice-versa to allow traffic from lower to higher an ACL must be applied to the lower interface.
To allow VPN traffic regardless of access-lists use: sysopt connection permit-vpn
If you want to control VPN traffic with VPN filter use:
group-policy gp_test attributes
 vpn-filter value acl_vpn_filter


Access-groups
To apply an access-list use access-group command:
access-group acl_inside_in in interface inside
access-group acl_inside_out out interface inside
access-group acl_global global
 

 

PS Need to add acl and acl-group, global acl

Understanding Main Fibre Optic Types/Specification

Category-CableColor:

1) Single Mode (SM) / OM1, OM2 (Optical Multimode) - long distances (2km and more) - yellow

2) Multi Mode (MM) - short distances (100Mbit 2km / 1Gbit 1km / 10Gbit - 550m) - orange

3) OM3 / OM4 (laser optimized) - aqua

Connector - also can serve as attenuator (`signal reducer`):

1) SC (Square Connector)

2) LC (Lucent/Little Connector) - smaller form of SC

3) FC (Fibre Connector) - round, threaded (like screw) connector for high-vibration environments

Attenuation needed because strong signal can damage photo-receiver

Mode:

1) SX (Simplex) - one cable (no pair)

2) DX (Duplex) - pair of joint cables

Polishing used to reduce insertion loss (different polishing types can work together but insertion loss increases):

1) PC (Physical Connector) / SPC (Super PC) / UPC (Ultra PC) - OM1 - beige or grey connector / OM2 - black connector / OM3,OM4 - aqua connector / SM - blue

2) APC (Angled PC) - SM - green connector
Let's use above info to determine cable specification:
LC - Lucent
SM - Single Mode (blue connector)
SX - Simplex (one cable)

ASA ASDM Java Web Start problem

Today I was trying to install new ASA 5512-X I have Java installed on my CentOS 7 but when trying to enter `https://192.168.1.1/admin/public/index.html` for downloading ASDM this `web-message` appeared:

Tried sequence of below and succeded:
Right click on `Java Web Start is required to run ASDM, but it is not installed on this computer.` and select `Inspect`

left click on `display: none;>` (<div id="jws_installed" style="display: none;">) and change `none` to `inline`
left click on `isplay: inline;>` (<div id="jws_not_installed" style="display: inline;">) and change `inline` to `none`

Now we can see changed ASA `web-menu`:

Click on `Run ASDM`, then copy downloaded `asdm.jnlp` file into desired directory and that's it...

How to access BIOS and Boot order (Fujitsu / HP servers)

Fujitsu:
1) BIOS -> F2
2) RAID -> set verbose in BIOS, then will appear while booting up
3) Boot order -> F12

HP:
1) BIOS -> F9
2) RAID -> F8
3) ILo -> F8 immediately after server is beginning to start up
4) Boot order -> F11

Cisco ASA can't backup problem.
Backing-up with Python script.

Recently I faced a problem - I can't backup ASA neither via ASDM (ASDM is not working) nor via SCP or FTP (variously errors are appearing). So the only way to backup is to make (because we need to backup VPN keys etc.):
more system:running-config
But it's very annoying to hit Enter many times (my config is all about  10000 lines). So I'll use small Python script (prerequisite is Python pexpect).  
#!/usr/bin/python
# -*- coding: utf-8 -*-

import pexpect
import time

asa_ip = '10.10.1.1'
asa_pass = '123456'
asa_enable_pass = '654321'
asa_name = 'test-asa'
connection_string = 'ssh -1 cisco@' + asa_ip
asa = pexpect.spawn(connection_string)
asa.logfile = open("asa_backup.log", "w")

def sender(exp, resp):
 asa.expect(exp)
 asa.send(resp)
 asa.sendcontrol('m') #send 'Enter' key

sender('password:', asa_pass)
sender('>', 'enable')
sender('Password:', asa_enable_pass)
sender('#', 'conf t')
sender('#', 'terminal pager 15000')
sender('#', 'more system:running-config')
'''ensure that entire config is showed
'we can do without that but
you'll face problem if you have '#' character in your
running-config
'''
asa.expect(': end')
backup = asa.before.split('\n')
sender('#', 'end')
sender('#', 'q')
dt = time.strftime("%Y-%m-%d")
backup_file = asa_name + "-asa_backup-" + dt
with open(backup_file, "w") as f:
 for line_id, line in enumerate(backup):
  if line_id not in (0, len(backup)-1, len(backup)):
   f.write(line)
 f.write(': end')
print "OK"

[admin@gfs1 Desktop]$   chmod +x asa_back.py
[admin@gfs1 Desktop]$   ./asa_back.py
[admin@gfs1 Desktop]$ ./asa_back.py
OK
[admin@gfs1 Desktop]$ ll asa_back*
-rwxrwxr-x 1 admin admin    955 Feb  2 18:20 asa_back.py
-rw-rw-r-- 1 admin admin 449936 Feb  2 18:21 asa_backup-2017-02-02
-rw-rw-r-- 1 admin admin 450298 Feb  2 18:21 asa_backup.log