Bash function and logging for an rsync backup script
To synchronize multiple folders via rsync, the individual folders for the sync can be written into a bash file as follows, so far so simple:
#!/bin/bash
rsync -av --delete /folder1/ /backup/folder1
rsync -av --delete /folder2/ /backup/folder2
rsync -av --delete /folder3/ /backup/folder3
rsync -av --delete /folder4/ /backup/folder4
To allow the status of the operation to be logged in a text file, I have adapted the lines as follows:
#!/bin/bash
printf "`date '+%Y%m%d %H%M'` /folder1 `rsync -av --delete /folder1/ /backup/folder1 --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished `date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
printf "`date '+%Y%m%d %H%M'` /folder2 `rsync -av --delete /folder2/ /backup/folder2 --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished `date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
printf "`date '+%Y%m%d %H%M'` /folder3 `rsync -av --delete /folder3/ /backup/folder3 --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished `date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
printf "`date '+%Y%m%d %H%M'` /folder4 `rsync -av --delete /folder4/ /backup/folder4 --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished `date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
The --stats parameter allows to write a summary of the sync operation to a log file;
head -n 2000 limits the logfile to 2000 lines and writes new entries to the beginning of the logfile via ">".
So that I don't have to change the folder names in each line several times, these can be replaced with variables:
#!/bin/bash
FOLDER=/folder1
printf "`date '+%Y%m%d %H%M'` $FOLDER `rsync -av --delete $FOLDER/ /backup$FOLDER --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished \n`date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
FOLDER=/folder2
printf "`date '+%Y%m%d %H%M'` $FOLDER `rsync -av --delete $FOLDER/ /backup$FOLDER --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished \n`date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
FOLDER=/folder3
printf "`date '+%Y%m%d %H%M'` $FOLDER `rsync -av --delete $FOLDER/ /backup$FOLDER --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished \n`date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
FOLDER=/folder4
printf "`date '+%Y%m%d %H%M'` $FOLDER `rsync -av --delete $FOLDER/ /backup$FOLDER --exclude .snapshots* --exclude .Trash* --stats | sed '0,/^$/d'` finished \n`date '+%Y%m%d %H%M'` \n\n`head -n 2000 /var/log/rsync.log`" > /var/log/rsync.log
After I also copy the command line multiple times with this variant, it can be swapped out into a function. On this occasion I split the one-liner with "\" at the end of the line into several lines:
#!/bin/bash
backup () {
printf "`date '+%d.%m.%Y %H:%M'` | $1 \n---------------------------------------\n`\
rsync -av --delete $1/ /backup$1 --exclude .snapshots* --exclude .Trash* --stats | \
sed '0,/^$/d'`\nfinished: `date '+%d.%m.%Y %H:%M'` \n---------------------------------------\n\n`\
head -n 2000 /var/log/rsync.txt`" > /var/log/rsync.txt
}
backup /folder1
backup /folder2
backup /folder3
backup /folder4
So far so clear, but to make the log output more readable, I split the log lines into variables. This way you can see a summary of the backup process at the beginning of the logfile, including the runtime of the backup folders and the update of the summary after the script is finished:
#!/bin/bash
LOGFILE="/var/log/rsync.txt"
STARTTIME="`date '+%d.%m.%y %H:%M'`"
START=`date +%s`
THISLOGHEAD="______________________________________\n\
|\n\
| Script: $0\n\
| Hostname: `hostname`\n\
| Started: $STARTTIME\n"
LOGCONTENT="`head -n 2000 $LOGFILE`"
THISLOG=""
SUMMARY=""
backup () {
local FSTARTTIME="`date '+%H:%M'`"
local FSTART=`date +%s`
printf "$THISLOGHEAD|\n\n\
---------------------------------------\n\
| Summary:\n\
---------------------------------------\n\
$SUMMARY\n\
`date '+%H:%M'`-now: $1 -> $2\n\n\
---------------------------------------\n\
| Details:\
\n---------------------------------------\n\
$THISLOG\n$LOGCONTENT" > $LOGFILE
THISLOG="$THISLOG\n`date '+%d.%m.%y %H:%M'` | $1 \n->$2\n---------------------------------------\n`\
rsync -av --delete $1 $2 --exclude .snapshots* --exclude .Trash* --stats | \
sed '0,/^$/d'`\nfinished: `date '+%d.%m.%y %H:%M'` \n---------------------------------------\n\n"
printf "$THISLOGHEAD\n$THISLOG\n$LOGCONTENT" > $LOGFILE
local FSECONDS="$((`date +%s`-$FSTART))"
SUMMARY="$SUMMARY \n$FSTARTTIME-`date '+%H:%M'` (`date -d@$FSECONDS -u +%H:%M:%S`): $1 -> $2"
}
backup /folder1 /backup
backup /folder2 /backup
backup /folder3 /backup
backup /folder4 /backup
#Finish Script: Runtime-Information and Final-Summary:
SECONDS="$((`date +%s`-$START))"
printf "$THISLOGHEAD\
| Finished: `date '+%d.%m.%y %H:%M'`\n\
| Runtime: `date -d@$SECONDS -u +%H:%M:%S`\n|\n\n\
---------------------------------------\n\
| Summary:\n\
---------------------------------------\n\
$SUMMARY\n\n\
---------------------------------------\n\
| Details:\n\
---------------------------------------\n\
$THISLOG\n\
|______________________________________\n\
$LOGCONTENT" > $LOGFILE
The variable $THISLOG collects the log entries during the runtime of the script, $SUMMARY is used for the summary and printed with "printf ... > $LOGFILE" the logfile is rewritten at every update.
For the header of the script the variable $THISLOGHEAD is used and the original log content is appended by the variable $LOGCONTENT. The destination can be specified as the second parameter in this example. The output to the text file looks like this with my backup folders:
______________________________________
|
| Script: ./rsync-backup.sh
| Hostname: sox
| Started: 06.08.21 09:33
|
---------------------------------------
| Summary:
---------------------------------------
09:33-09:33 (00:00:01): /scripts -> /media/dreambox/backup/scripts-nas
09:33-09:41 (00:07:57): /daten/Dokumente -> /media/dreambox/backup/Dokumente
09:41-09:42 (00:00:26): /daten/PhotoVideo -> /media/dreambox/backup/PhotoVideo
09:42-09:47 (00:05:47): /docker -> /media/dreambox/backup/docker
...
---------------------------------------
| Details:
---------------------------------------
06.08.21 09:33 | /scripts
->/media/dreambox/backup/scripts-nas
---------------------------------------
Number of files: 23 (reg: 20, dir: 3)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 4
Total file size: 26,129 bytes
Total transferred file size: 3,347 bytes
Literal data: 3,347 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 4,377
Total bytes received: 109
sent 4,377 bytes received 109 bytes 2,990.67 bytes/sec
total size is 26,129 speedup is 5.82
finished: 06.08.21 09:33
---------------------------------------
06.08.21 09:33 | /daten/Dokumente
->/media/dreambox/backup/Dokumente
---------------------------------------
Number of files: 353,503 (reg: 324,561, dir: 28,891, link: 51)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 83
Total file size: 204,805,207,475 bytes
Total transferred file size: 449,566,337 bytes
Literal data: 449,566,337 bytes
Matched data: 0 bytes
File list size: 3,143,254
File list generation time: 0.038 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 459,277,206
Total bytes received: 31,776
sent 459,277,206 bytes received 31,776 bytes 961,903.63 bytes/sec
total size is 204,805,207,475 speedup is 445.90
finished: 06.08.21 09:41
---------------------------------------
...
{{percentage}} % positive