I finally got some online storage to backup my home server to, and I can finally sleep easy when my house inevitably goes up in flames. My initial push has been with rclone using the sync command. I plan on running a daily or weekly push.
I’ve gone with Hetzner and they have a variety of options on how to connect to their storage box as they call it. They have an option for automatic snapshotting of, as I understand it, the entire storage box that you can then recreate.
So my thoughts are around rclones sync and copy commands, and the storage snapshots.
Copy) If I use copy, it won’t clear up deleted files automatically that I actually want gone.
Sync) With sync I’m worried that say a drive failed (my set up is a bit wonky currently until I can build a proper RAID), it will do a sync, see an entire directory is gone, and go ahead and delete it.
Snapshots) Here I’m worried that any new data that’s been added in-between the snap shot and a server failure would be lost. And I don’t you can retrieve partial data from a snapshot.
What I have in mind is to use rclone’s copy command, but then somehow have a retention period on files that no longer exist on my server. So a deleted file might exist in my storage bucket for a month before being removed from there as well.
How do you guys manage this? Are there any ready made solutions, or do I need to whip up a script that goes through and looks at the last modified dates and removes the old ones?
Use duplicati, or if you want command line, use borg.
I have been trying out duplicati for backups the last few weeks and so far it’s doing what I need and I think would offer the features you are looking for.
Yeah that looks cool. I will give it a closer look tomorrow :)
I think you need something like restic with a retention policy
https://restic.readthedocs.io/en/stable/060_forget.html#removing-snapshots-according-to-a-policy
--keep-{hourly,daily,weekly,monthly,yearly}other solutions that implement similar policies are
kopiaandrusticthe advantage of using an off the shelf solution is that it’s almost certainly more reliable than what anyone can come up with in a few hours, and, it works with incremental backups, so your space requirements are drastically reduced depending how often you run it.
Restic is great, and the de-duplication between snapshots is amazingly good. Same content in different files (e.g. tar files of linux systems) take very little space like magic). Backrest is a nice web frontend for it.
Note that you should use some retention features of your provider to manage the risk of ransomware deleting your backups.
you might want to add borg, restic and - more recently - plakar to that list
Scripting it isn’t that tricky.
I’m old and use rsync, with mtime and not(!) operands and I’m able to keep 7 daily, 4 weekly, and 4 monthly backups. Runs every day, the rest of the backups are pruned in this rolling window.
