My Backup Script: Rsync-Based With Orgmode-Friendly Summary and Desktop Notifications

Update 2022-01-27: section "My Current Backup Strategy"

I'd like to share my current GNU/Linux backup shell script since it exceeded the level of being trivial slightly and has some nice properties you might want to have as well:

• Able to handle and detect different backup disks with different configurations.
• Using rsnapshot (which is a wrapper to rsync) resulting in navigationable backup results.
• Target disk is LUKS encrypted (setup not explained here).
• A backup run summary is not only printed to the console output but also added to your Org mode inbox file. This summary can then be easily refiled to its final destination.
• Desktop notifications via notify-send when backup run is finished and target disk is unmounted.

Today, I ran the backup script myself and this is what popped up in my inbox.org. By executing the babel script, this heading got moved to its final destination.

 ** <2020-08-29 Sat 11:06>--<2020-08-29 Sat 11:17> vkbackup: 4TBbackup took 00:10:50 (HH:MM:SS)

 #+BEGIN_SRC emacs-lisp
 (my-org-refile "misc.org" "Backup: My 4TB backup")
 #+END_SRC

 - logfile: [[file:/home/user/log/vkbackup-4TBbackup.log][/home/user/log/vkbackup-4TBbackup.log]]
 - Backup target UUID: ababababab-1234-4b39-8233-9182737465

 - "df -h" before backup:

   | Filesystem                 | Size | Used | Avail | Use% | Mounted        |
   |----------------------------+------+------+-------+------+----------------|
   | udev                       | 7.8G | 0    | 7.8G  |   0% | /dev           |
   | tmpfs                      | 1.6G | 164M | 1.5G  |  11% | /run           |
   | /dev/mapper/host-root-luks | 101G | 35G  | 62G   |  36% | /              |
   | tmpfs                      | 7.8G | 553M | 7.3G  |   7% | /dev/shm       |
   | /dev/sda1                  | 965M | 170M | 741M  |  19% | /boot          |
   | /dev/mapper/hostLVluks     | 7.3T | 6.7T | 363G  |  95% | /home/user     |
   | /dev/dm-3                  | 3.6T | 1.3T | 2.2T  |  38% | /mnt/backup    |

 - "df -h" after backup:

   | Filesystem                 | Size | Used | Avail | Use% | Mounted        |
   |----------------------------+------+------+-------+------+----------------|
   | udev                       | 7.8G | 0    | 7.8G  |   0% | /dev           |
   | tmpfs                      | 1.6G | 164M | 1.5G  |  11% | /run           |
   | /dev/mapper/host-root-luks | 101G | 35G  | 62G   |  36% | /              |
   | tmpfs                      | 7.8G | 545M | 7.3G  |   7% | /dev/shm       |
   | /dev/sda1                  | 965M | 170M | 741M  |  19% | /boot          |
   | /dev/mapper/hostLVluks     | 7.3T | 6.7T | 363G  |  95% | /home/user     |
   | /dev/dm-3                  | 3.6T | 1.3T | 2.2T  |  38% | /mnt/backup    |

 File content of "/home/user/log/vkbackup-4TBbackup.log":
 #+BEGIN_EXAMPLE
 [2020-08-29T11:07:02] /usr/bin/rsnapshot -c /home/user/archive/backup/host/rsnapshot_4TBbackup.conf alpha: started
 [2020-08-29T11:07:02] echo 15273 > /var/run/rsnapshot.pid
 [2020-08-29T11:07:02] /bin/rm -rf /mnt/backup/host/alpha.2/
 [2020-08-29T11:09:14] mv /mnt/backup/host/alpha.1/ /mnt/backup/host/alpha.2/
 [2020-08-29T11:09:14] /bin/cp -al /mnt/backup/host/alpha.0 /mnt/backup/host/alpha.1
 [2020-08-29T11:11:28] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
                       --exclude-from=/home/user/archive/backup/host/rsnapshot_exclude_file_4TBbackup.txt \
                       /home/user/host/ /mnt/backup/host/alpha.0/host/
 [2020-08-29T11:17:52] touch /mnt/backup/host/alpha.0/
 [2020-08-29T11:17:52] rm -f /var/run/rsnapshot.pid
 [2020-08-29T11:17:52] /usr/bin/rsnapshot -c /home/user/archive/backup/host/rsnapshot_4TBbackup.conf \
                       alpha: completed successfully
 #+END_EXAMPLE	  

You can find the backup script on GitHub. Please do read the instructions at the top of the script carefully and adapt the script according to your situation.

At the moment, my backup strategy is like the following:

• I use my biggest 3,5" hard disk (HDD) as my main disk, conntected via USB 3 to my small home computer: Brix GB-BRR7H-4800 AMD Ryzen R7-4800U as of 2022-01-27.
• My second biggest HDD is my monthly backup. For size restrictions, I have to ignore large sets of my data, mostly ephemeral video files.
• My third biggest HDD is my off-site backup which I keep at the office or I ask somebody I know to take care for it and bring it to me on occasional visits. This results in approximately six months interval.
• My forth biggest HDD is my weekly backup where I have to concentrate on the most important files. However, this backup doen't take too long and I tend to keep more versions of older files.

I bought a sharkoon HDD adapter (SATA QuickPort USB 3.1 Type C) so that I don't have to use expensive external HDD cases for each HDD. With this neat adapter, I may plug in any HDD like a ROM cartridge within seconds and start my backup script from above. It detects the UUID automatically and runs the configured rsnapshot setup with the associated exclude lists for it.

My Orgdown files are my most important data. They are synchronized among all of my notebook and desktop computers. Therefore, those connected computers might serve as a short-term backup in case my main computer should have urgent hardware issues. Remember: a synchronization solution never replaces a normal backup because you might delete important files without realizing. Therefore, you always need a backup concept.

I was using rsync-based backup for decades. In recent months, I tested restic which has some really nice features and an extremely efficient way of saving backup disk space.

Unfortunately, restic is not able to handle out-of-disk situations properly, ending up in broken snapshots that need to be removed. Much worse, to get back to a working backup setup you most likely end up with starting from scratch, deleting your previous backup snapshots.

If you don't have to worry about disk space of the backup disk, you're fine with restic. If you follow the approach that I'm using, you should not use restic at all.

This is my personal backup strategy with a main disk being larger than my backup disk:

• I run my backups with a minimal ignore list to backup as much as possible on the smaller backup disk.
• When the backup fails because of an out of disk situation:
  1. I add more paths to the ignore list.
  2. I clean those paths from the backup data, freeing up space.
     • You can't do this with restic in an efficient way. At least without being a real wizard with knowledge of restic internals.
  3. I re-run the backup.
  4. If it fails, I start this sub-process from number one. If not, I got a successful backup.
• Backup finished successfully.

After testing and failing with restic, I'm happy to be back to a simple rsync-based approach. It may be less efficient when it comes to storage formats but it's working with simple out-of-the-box GNU/Linux features such as hardlinks. I know how to handle situations with those standard features very well. I don't want to be a restic wizard in order to understand how to handle situations where restic does not support my requirements by default.

Therefore, I do not have any urges to test yet another cool backup solution at the moment.