bash rotation script

Question

I have a directory structure as follows:

backup-2018-01-12 backup-2018-01-13 backup-2018-01-14 backup-2018-01-15 backup-2018-01-16 backup-2018-01-17 backup-2018-01-18 backup-2018-01-19 backup-2018-01-20 backup-2018-01-21 backup-2018-01-22 backup-2018-01-23 backup-2018-01-24 backup-2018-01-25 backup-2018-01-26 backup-2018-01-27 backup-2018-01-28 backup-2018-01-29 backup-2018-01-30 backup-2018-01-31 backup-2018-02-01 backup-2018-02-02 backup-2018-02-03 backup-2018-02-04 backup-2018-02-05 backup-2018-02-06 backup-2018-02-07 backup-2018-02-08 backup-2018-02-09 backup-2018-02-10 backup-2018-02-11 backup-2018-02-12 backup-2018-02-13 backup-2018-02-14 backup-2018-02-15

How can I use a script to save the latest 7 days and the the last backup of each week for 4 weeks long

E.g

So I keep

backup-2018-01-25 <-- this is 3 waeks from now and so on. backup-2018-02-02 <-- this is 2 weeks earlier from now backup-2018-02-09 backup-2018-02-10 backup-2018-02-11 backup-2018-02-12 backup-2018-02-13 backup-2018-02-14 backup-2018-02-15

So I tried

find -type d -name 'backup-*' -mtime +7 -exec rm -v {} \;

and this does keep the latest 7 but removes everything older than 7 days.

Some use dedicated backup software for these types of tasks (i.e. backing up data and keeping backups of certain ages), like borgbackup or borgmatic (which uses borgbackup). — Kusalananda, CommentedFeb 15, 2018 at 13:14
I see not obvious tools for the jobs, facing similar problem i wrote a c program. — Archemar, CommentedFeb 15, 2018 at 13:22
This is the first hit on Google for this type of backup retention which is my primary requirement. However, I'm looking for something with a GUI, unless I'm forced to use a script so I created a post in SE-software-reqs here: softwarerecs.stackexchange.com/questions/84066/… — alchemy, CommentedSep 22, 2022 at 2:19

Chris Davies · Accepted Answer · 2018-02-15 14:50:44Z

I haven't been able to find a straightforward solution. The easiest way I found to solve it was to use a "keep" list rather than a "remove" list. Then I could add the last N backups to the list, and the newest of each week's range for M weeks. (In my own code, written some time ago, I extended to months, quarters, and years.) Obviously there will be some overlap but because it's a "keep" list that doesn't matter. Then you delete everything that isn't in the list - having first checked the list isn't empty.

#!/bin/bash # dirs=( backup-2018-01-12 backup-2018-01-13 backup-2018-01-14 backup-2018-01-15 backup-2018-01-16 backup-2018-01-17 backup-2018-01-18 backup-2018-01-19 backup-2018-01-20 backup-2018-01-21 backup-2018-01-22 backup-2018-01-23 backup-2018-01-24 backup-2018-01-25 backup-2018-01-26 backup-2018-01-27 backup-2018-01-28 backup-2018-01-29 backup-2018-01-30 backup-2018-01-31 backup-2018-02-01 backup-2018-02-02 backup-2018-02-03 backup-2018-02-04 backup-2018-02-05 backup-2018-02-06 backup-2018-02-07 backup-2018-02-08 backup-2018-02-09 backup-2018-02-10 backup-2018-02-11 backup-2018-02-12 backup-2018-02-13 backup-2018-02-14 backup-2018-02-15 ) declare -A list declare -A keep #################################################################################### # Go # prefix="backup-" today=$(date +%Y-%m-%d) # Populate list from set of directories # for item in "${dirs[@]}" do list[$item]="$item" done # Build keep list from business criteria # for dayback in {0..6} do dateback=$(date --date "$today - $dayback day" +%Y-%m-%d) keep[$prefix$dateback]=1 done for weekback in {0..3} do dateback1=$(date --date "$today - $weekback week" +%Y-%m-%d) dateback2=$(date --date "$today - $((weekback + 1)) week" +%Y-%m-%d) # Look for newest in range # newest= dateback="$dateback1" while [[ $dateback != $dateback2 ]] do [[ -n ${list[$prefix$dateback]} ]] && newest=$dateback && break dateback=$(date --date "$dateback - 1 day" +%Y-%m-%d) done [[ -n $newest ]] && keep[$prefix$newest]=1 done # We now have a keep list, so delete anything not in that list # for item in "${list[@]}" do [[ -n ${keep[$item]} ]] && echo "KEEP $item" || echo "DELETE $item" done # All done # exit 0

When you're happy, change or extend the echo statements towards the end to perform the actual deletion.

I have used a similar solution in the past, where I kept directories for daily, weekly, and monthly backups, and just moved files around, and kept various numbers of files in each directory. I like the idea of a keep list, though. — Tim Kennedy, CommentedFeb 15, 2018 at 15:52
@roaima awesome! It almost working. What if directory is /some/path/backup-* how to dael with this? — holasz, CommentedFeb 17, 2018 at 18:14
@roaima , How can i make this work when dir is like dirs= ( /var/lib/somepath/data/*/*/*/backups/backup-*) where under data there are different directories each with backups/backup-* . This work when the script is located in the same location as backups-* the above. — holasz, CommentedFeb 21, 2018 at 20:35
@holasz I'd discard the concept of a $prefix entirely, and instead just build a set of dates to keep. I'd then compare the date component of the backup directory paths to the set of dates I'd generated, deleting the directories for which there was no match. I suspect it's a harder problem to solve than the one you described in your question, though. — Chris Davies, CommentedFeb 21, 2018 at 23:20

Marc · Accepted Answer · 2018-02-15 14:52:17Z

Whats about this?

for i in $(find . -maxdepth 1); do if ! (( $((($(date +%s) - $(date +%s -r $i )) / 86400)) % 7)); then echo $i # Print out the folder. Can be replaced with your action fi done

or as oneliner:

for i in $(find . -maxdepth 1); do if ! (( $((($(date +%s) - $(date +%s -r $i )) / 86400)) % 7)); then echo "$i $i_weekly" ;fi; done

This should just print all folders which should be 7,14,21,28.... days old. Instead of "echo" just make a mv or what ever...

may be you need to change the path of the find command. The "." means use the current working directory, so change in the parent dir of your backups first. — Marc, CommentedFeb 19, 2018 at 10:06

mac · Accepted Answer · 2021-09-16 15:07:53Z

0

Here is an example script to rotate backups but keep last 7 days, 12 monthes backup-*-*-01 and 10 years backup-*-01-01

all=$(ls backup-*) days=$(ls backup-* | tail -n 7) monthes=$(ls backup-*-01 | tail -n 12) years=$(ls backup-*-01-01 | tail -n 10) keep=" $days $monthes $years " for item in $all do if ! [[ "$keep" =~ "$item" ]] then echo "$item will be deleted" # rm $item fi done

edited Sep 16, 2021 at 15:07

answered Sep 15, 2021 at 9:27

mac

1012 bronze badges

2
Don't use ls - it isn't needed or wanted. Try running this BUT replacing the 'rm $item' with 'echo $item'.
– Jeremy Boden
CommentedSep 15, 2021 at 14:38
@JeremyBoden thank you for the comment, I have replaced unsafe rm with an echo example, also I'm not sure what did you mean by saying that ls isn't needed, can you please describe it a little bit more, I have tried something like all=$(backup-*); echo $all with no luck
– mac
CommentedSep 16, 2021 at 15:10
You are probably OK with ls in this case since you know that all names won't have spaces etc in them. Also you (hopefully) will get filenames in date sequence (which would allow the tail commands to work properly). I always try to avoid using ls in scripts...
– Jeremy Boden
CommentedSep 16, 2021 at 23:10

Add a comment |

Stack Exchange Network

bash rotation script

3 Answers 3

You must log in to answer this question.

Hot Network Questions

bash rotation script

3 Answers 3

You must log in to answer this question.

Related

Hot Network Questions