linux n00b

Remote Network Backup

Posted in Administration, Desktop, Howto by downforce on April 3, 2007

My wife & I have a 7 week newborn and we have been taking photos like crazy, thanks to a big 1GB memory card. Very very quickly we are getting a lot of photo’s, like another 2gb+ in the first 7 weeks and being digital, it’s so easy to loose them, either through HDD failures [unlikely but possible], accidental deletion [wife learning Linux desktop – very possible!]; or my worst fear, fire or theft – someone picking up my PC and walking out the door or my house burning down!

I have almost a terabyte of hard drive space (actually 948GB) scattered through my 3 machines. Only 750GB is in my file server in LVM and only 400GB-ish is currently used. I made the realisation very quickly that I’m never going to back it all up, but I do have at my disposal, thanks to a kind brother-in-law, a DDS20/40 tape drive. Not only that, I don’t want/need a full enterprise backup solution. If any of the PC’s OS die/gets severely corrupted, I’ll reinstall from scratch. It’s an inconvenience, but nothing I have running is ‘mission critical’.

So I’ve had to decide what I’m going to backup. Basically, all I care about is my documents (about 1GB worth), my photo’s (now 8GB), both are kept on my desktop PC and the config files of my server (/etc – about 200MB (I think)). All my music and videos are going to have to be sacrificed if the worst was to happen.

I’ve decided to implement a two-fold backup strategy. I have my originals on my desktop PC, I regularly copy my docs/photo’s to a specific backup share on my file server then less regularly palm it off on to a tape. At the moment it’s all done manually, I manually copy the files, I manually run the tape script (synbak).

To the point – I wanted to do it automatically!

Firstly, I wanted to setup an automated way of copying my local docs/photos to the file server. After a bit of Googling I found the article Backup for the Home Network from Linux Gazette. I wanted a script that did incremental backups and could handle doing it over the network. This fulfilled both requirements! This is how I set it up:

  • I grabbed their backup script, run-backup and saved it to my script directory, ~/scripts. This is a directory under my home drive where I put all my scripts. Depending on what it does, sometimes I’ll create a symlink to /usr/local/bin, but in this case it’s a personal backup so I’ll skip the step. **
  • Make the script executable, just for me though!
      chmod 700 run-backup
  • I ensure I have my file server being mounted on boot-up in /etc/fstab to my backup share, /data/backups. See below about the section on mounting with samba.
  • Tweak the run-backup script with my settings.
    • I needed to create the last-full directory so:
        sudo mkdir /data/backups/last-full
    • Double check tar is in the right place:
        which tar
    • I changed the Full Backup day to a Monday, changed $DOW = "Sun" to $DOW = "Mon". We tend to take most of our photo’s on the weekend, so this was a better choice, plus being a desktop PC it gets used more on the weekends – I don’t want it slowing down while it’s backing up.
  • Next I set up a cronjob which runs this on a daily basis at 4pm. My home PC is almost always on, at least it’s always on when I’m awake. I did this as my user, not a root cronjob:
      crontab –e
  • I added:
      * 16 * * /home/user/scripts/run-backup
  • Done!

Just to make sure the script is working, I run it manually. Depending on the amount of files it could take awhile. My 9 GB worth takes about 25-30min.

Problems with Samba!
When I was testing, the script kept dieing at about 2GB. Upon closer examination, it died at exactly 2147483647bytes each time for 3 runs. At the time I didn’t have time to examine what was wrong, but during my lunchbreak I was reading my Linux news and lucked up the article Using external file devices in Linux: Climbing the “mount” command.

At the end of the article, the author, dcroxton, had the same problem, there is a limitation of samba in that it only handles 2GB files by default. I’d found the solution without even looking for it!

With a quick add of ,lfs to the end of my /etc/fstab, I was back in business! I ran the script and all 9GB went across fine. Just to be sure, I extracted the files out again and checked them and they were all fine and dandy!

The next article is going to be on the synbak script and backing up it all up to tape.

** The reason for this is I’ve broken a few installations. As /home is a separate partition, when remount it, all my scripts are back! I’ll post what I have in there soon. There are some nice little ‘helper’ scripts.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: