Linux tar incremental backup & restore

1. Backup

In this post I'll describe steps to use tar for incremental backup of files.

Incremental backups needed when it's lack of free space on your backup device so you have full-backup and incremental backup which contains changes/differences.

To make incremental backups tar uses `.snar` meta-files which contains changes since last backup. We'll use following tar options for backup (-vcSpzg):

1. -c OR --create - tells that we are creating new archive.
2. -v | --verbose - useful if you want log files.
3. -S OR --sparse - sparse files are files having holes (the section of the files' contents which was never written, the contents of a hole reads as all zeroes). Many file systems not actually allocating storage for hole but saving length of the hole so without that option tar archive will be longer than the original. Using this option makes tar to recognize sparse files and save only actual date. When restoring tar will add holes to such files at the needed places.
4. -p OR --preserve-permissions - we tell tar to save all permissions for every file.
5. -z OR --gzip - use compression.
6. -g OR --listed-incremental - show which meta-data file to use, this options show that we make backup and show's it's level. 
7. -f OR --file - backup file name.

The core concept is:

1. if snar file in `-g` option is not existing - make full backup and create snar file containing list of files backuped
2. if snar file exists - make incremental backup not backuping files listed in snar file

We'll have 7 files:

1.  for Monday - dir-dump1.sh
2.  for Tuesday - dir-dump2.sh
3.  for Wednesday - dir-dump3.sh
4.  for Thursday - dir-dump4.sh
5.  for Friday - dir-dump5.sh
6.  for Saturday - dir-dump6.sh
7.  for Sunday - dir-dump7.sh

So for example if we want to make full backup on Monday:
cat dir-dump1.sh
#! /bin/bash
#remember core concept (1) - we removing snar file for full-backup day
/bin/rm -f /mnt/backup/snar-dir/monday0.snar

#monday0.snar backup file will be created during full-backup
#$(date +%W) returns number of the week of the year
/bin/tar -cvSpzg /mnt/backup/snar-dir/monday0.snar -f /mnt/backup/$(date +%W)-monday.gz   /var/lib/recordings

For Tuesday we make incremental backup with file differing from monday backup:
cat dir-dump2.sh
#! /bin/bash
#core concept (2) - specified snar file will be used for incremental backup
/bin/cp -f /mnt/backup/snar-dir/monday0.snar /mnt/backup/snar-dir/tuesday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/tuesday.snar -f /mnt/backup/$(date +%W)-tuesday.gz   /var/lib/recordings


For Wednesday, Thursday, Friday, Saturday & Sunday we make the same - renaming snar file of the previous day:

cat dir-dump3.sh
#! /bin/bash
/bin/mv -f /mnt/backup/snar-dir/tuesday.snar /mnt/backup/snar-dir/wednesday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/wednesday.snar -f /mnt/backup/$(date +%W)-wednesday.gz   /var/lib/recordings

cat dir-dump4.sh
#! /bin/bash
/bin/cp -f /mnt/backup/snar-dir/wednesday.snar /mnt/backup/snar-dir/thursday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/thursday.snar -f /mnt/backup/$(date +%W)-thursday.gz   /var/lib/recordings

cat dir-dump5.sh
#! /bin/bash
/bin/cp -f /mnt/backup/snar-dir/thursday.snar /mnt/backup/snar-dir/friday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/friday.snar -f /mnt/backup/$(date +%W)-friday.gz   /var/lib/recordings

cat dir-dump6.sh
#! /bin/bash
/bin/cp -f /mnt/backup/snar-dir/friday.snar /mnt/backup/snar-dir/saturday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/saturday.snar -f /mnt/backup/$(date +%W)-saturday.gz   /var/lib/recordings

cat dir-dump7.sh
#! /bin/bash
/bin/cp -f /mnt/backup/snar-dir/saturday.snar /mnt/backup/snar-dir/sunday.snar
/bin/tar -cvSpzg /mnt/backup/snar-dir/sunday.snar -f /mnt/backup/$(date +%W)-sunday.gz /var/lib/recordings

Then add backup for every day to the crontab (redirecting all errors and output to the log file):
0 5 * * 1 /mnt/backup/dir-dump1.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 2 /mnt/backup/dir-dump2.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 3 /mnt/backup/dir-dump3.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 4 /mnt/backup/dir-dump4.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 5 /mnt/backup/dir-dump5.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 6 /mnt/backup/dir-dump6.sh 2>&1>> /mnt/backup/dir-dump.log
0 5 * * 7 /mnt/backup/dir-dump7.sh 2>&1>> /mnt/backup/dir-dump.log

If you understood core concepts you can do any mix of full and incremental backups, using snar files.

 

2. Restore

Meta-files not needed for restoring. They needed only for the incremental backup process.  To restore we first restore last full backup and then all incremental backups. If use our backup scheme as example, restore will be as (i.e. we restore files for 10th week of the year). We'll ise `/` in `--directory` option because tar saves full-path and showing `/var/lib/recordings` will result in having `/var/lib/recordings/var/lib/recordings/file_name`. For snar file we specify `/dev/null` it is necessary because option `-g` must be specified in order to keep-track on operations over files during incremental backup (deletion, creation, changing) and snar file option can't be empty if you specify `-g` option:

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-monday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-tuesday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-wednesday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-thursday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-friday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-saturday.gz  --directory=/

/bin/tar --extract -vSpzg /dev/null -f /mnt/backup/10-sunday.gz  --directory=/