Tuesday, November 30, 2010

FreeNAS Update: Rsync and Unison

I've been using FreeNAS since the beginning of April, 2010, and it certainly has met or exceeded my expectations in that time. It still serves its purpose well, however the way I have it set up has changed quite a bit since April. You can find my original post here (FreeNAS Review). I wanted to take the time to describe my first thoughts when creating my NAS file server, and how it has changed into its current form.

My NAS's main purpose is to be a file server and store movies, tv shows, and music to be used mostly by my AppleTV, but also with all computers on my network. I had started in June/July '09 with an external hard disk plugged into my Apple Airport Extreme router. The disk spun 24/7 and eventually became unreadable by September and I was mad. Luckily for me, I still had most of my data backed up on the gaming desktop. But the backup was taking most of my desktop's HDD space and so I wanted it all on its own. For that reason, I bought a Linksys NAS 200 that featured RAID1. I thought a second disk would at least keep my data safe if the disk crashed again. It was waaay too slow for my needs and so I built the FreeNAS server.

RAID1 vs. Rsync

FreeNAS was set up with the same RAID1 as the Linksys and it was much faster, so life was good. But as I talked about it more and more, RAID1 sounded less and less secure. Sure, I was safe from a hard disk failing but there were other things that could delete my collection still, and I'd be SOL. I could accidently type a command wrong and accidently delete the entire /Movies folder with a hit of the enter key, and that would happen on both drives instantly. Or the flash drive containing my FreeNAS config could fail and my software controller would be erased. Without the software controller settings, you can't access a RAID array (at least not easily, like at the user level). So for these reasons I backed up, wiped the RAID array away, and changed to a local rsync service.

Rsync runs once a day in the middle of the night, and basically mirrors the entire Media Drive onto the backup disk. I have until 3:00am to undue any accidental deletions or changes to the filesystem made during the day. If the media disk were to fail, I'd only lose a day's worth of new stuff which is usually a tv show or two. I'd rather re-download two tv shows than 1.5TB of movies and stuff!

Unison

In addition to backing up my media, the FreeNAS server acts as a remote location for backing up important files from my other computers, including my macbook pro(music), Gaming PC(Pictures), and Alison's files. The Macs simply use Time Machine to backup their files, and I could use Windows 7 Backup to store my desktop's files. But, I thought it'd be nice to sync my Pictures between the desktop and the laptops. That way, when I go home or do imagework, I always have the latest, up-to-date images on all the machines. Having them on different computers protects them against being lost if a disk dies, and unison can archive deleted files if you want it to which protects against accidental deletion. You can think of unison as two-way Rsync, in that any change you make will be backed up on the server, but if the server data changes, the changes will be sent to your computer as well.
To use Unison in FreeNAS, you only have to enable the service in the webadmin. You'll want SSH enabled as well. Everything else is done from the client computers. Install Unison (http://www.cis.upenn.edu/~bcpierce/unison/download.html), and create a private SSH key so you don't have to type in your SSH password (http://linuxproblem.org/art_9.html). Windows users will need to install Cygwin for the unix shell.

Next, create a preferences file on each computer you want to sync files to. Probably looking something like this:

batchpref.prf
Obviously, you'll want to change the username, IP address, and path to your own settings. You need to give it two directories to make the same. One local to the computer, and one on the remote disk or network location. "ignore" tells the config to ignore whatever files or regex you want. In my case, it's ignoring those annoying hidden windows and OSX files. You can set it to anything, even entire directories. And there are lots of other options you can do as well. See the User manual for all options (User Manual).

Once the config file is done, save it to the appropriate place.
In OS X, it's ~/Library/Application Support/Unison/batchpref.prf
In Windows (Cygwin), it'll be c:\cygwin\home\username\.unison\batchpref.prf
Linux is probably ~/.unison/batchpref.prf       (I dont have straight up linux, so I don't know for sure)

Now that the preferences are saved, you should be able to run unison from the command line. Type:
This tells the computer to run unison, in the text-only user interface, in automatic mode (doesnt ask you what to do in errors), and to use the preference file we just made. From here, you can run it as a cron job in OSX/Linux, or make a .bat file in windows and run it with Task Scheduler. I have it run on windows at startup, and once a day on the macs. You can also run it manually when you want to make sure you're up to date.

Now, I guess websites like dropbox offer similar capability, but you're either limited in filesize or you have to pay for it. Paying isn't a bad idea, since I suppose if my house burned down I'd still lose everything. But I find unison to be a great way to back up and syncronize easily and for free. Especially if your data is the "nice to have" but not the most important memories of your life.

I'll finish the post off with a newer pic of my home network. It's clickable to view it a little bigger. I still have other FreeNAS updates I can write about, but since Rsync and Unison are so closely related, I thought I'd give them their own post.

Name:

URL:

Smileys

Remember my personal information

Submit the word you see below: