Category
Featured
Best of DarkMercury.net
Tuesday, November 30, 2010
FreeNAS Update: Rsync and Unison
I've been using FreeNAS since the beginning of April, 2010, and it certainly has met or exceeded my expectations in that time. It still serves its purpose well, however the way I have it set up has changed quite a bit since April. You can find my original post here (FreeNAS Review). I wanted to take the time to describe my first thoughts when creating my NAS file server, and how it has changed into its current form.My NAS's main purpose is to be a file server and store movies, tv shows, and music to be used mostly by my AppleTV, but also with all computers on my network. I had started in June/July '09 with an external hard disk plugged into my Apple Airport Extreme router. The disk spun 24/7 and eventually became unreadable by September and I was mad. Luckily for me, I still had most of my data backed up on the gaming desktop. But the backup was taking most of my desktop's HDD space and so I wanted it all on its own. For that reason, I bought a Linksys NAS 200 that featured RAID1. I thought a second disk would at least keep my data safe if the disk crashed again. It was waaay too slow for my needs and so I built the FreeNAS server.
RAID1 vs. Rsync
FreeNAS was set up with the same RAID1 as the Linksys and it was much faster, so life was good. But as I talked about it more and more, RAID1 sounded less and less secure. Sure, I was safe from a hard disk failing but there were other things that could delete my collection still, and I'd be SOL. I could accidently type a command wrong and accidently delete the entire /Movies folder with a hit of the enter key, and that would happen on both drives instantly. Or the flash drive containing my FreeNAS config could fail and my software controller would be erased. Without the software controller settings, you can't access a RAID array (at least not easily, like at the user level). So for these reasons I backed up, wiped the RAID array away, and changed to a local rsync service.
|
|
Unison
In addition to backing up my media, the FreeNAS server acts as a remote location for backing up important files from my other computers, including my macbook pro(music), Gaming PC(Pictures), and Alison's files. The Macs simply use Time Machine to backup their files, and I could use Windows 7 Backup to store my desktop's files. But, I thought it'd be nice to sync my Pictures between the desktop and the laptops. That way, when I go home or do imagework, I always have the latest, up-to-date images on all the machines. Having them on different computers protects them against being lost if a disk dies, and unison can archive deleted files if you want it to which protects against accidental deletion. You can think of unison as two-way Rsync, in that any change you make will be backed up on the server, but if the server data changes, the changes will be sent to your computer as well.
Next, create a preferences file on each computer you want to sync files to. Probably looking something like this:
batchpref.prf
# Roots of the synchronization
root = /Users/username/Whatever/
root = ssh://username@192.168.X.XXX//mnt/Disk/PathTo/Wherever/
# Paths to synchronize
#path = current
#path = common
#path = .netscape/bookmarks.html
ignore = Name {._*,.DS_Store,Thumbs.db}
Obviously, you'll want to change the username, IP address, and path to your own settings. You need to give it two directories to make the same. One local to the computer, and one on the remote disk or network location. "ignore" tells the config to ignore whatever files or regex you want. In my case, it's ignoring those annoying hidden windows and OSX files. You can set it to anything, even entire directories. And there are lots of other options you can do as well. See the User manual for all options (User Manual).
Once the config file is done, save it to the appropriate place.
In OS X, it's ~/Library/Application Support/Unison/batchpref.prf
In Windows (Cygwin), it'll be c:\cygwin\home\username\.unison\batchpref.prf
Linux is probably ~/.unison/batchpref.prf (I dont have straight up linux, so I don't know for sure)
Now that the preferences are saved, you should be able to run unison from the command line. Type:
unison -ui text -auto -batch batchpref
This tells the computer to run unison, in the text-only user interface, in automatic mode (doesnt ask you what to do in errors), and to use the preference file we just made. From here, you can run it as a cron job in OSX/Linux, or make a .bat file in windows and run it with Task Scheduler. I have it run on windows at startup, and once a day on the macs. You can also run it manually when you want to make sure you're up to date.
Now, I guess websites like dropbox offer similar capability, but you're either limited in filesize or you have to pay for it. Paying isn't a bad idea, since I suppose if my house burned down I'd still lose everything. But I find unison to be a great way to back up and syncronize easily and for free. Especially if your data is the "nice to have" but not the most important memories of your life.
I'll finish the post off with a newer pic of my home network. It's clickable to view it a little bigger. I still have other FreeNAS updates I can write about, but since Rsync and Unison are so closely related, I thought I'd give them their own post.
Friday, October 29, 2010
Our Wedding Story
Thursday, April 29, 2010
FreeNAS Review
I promised in my last tech post that this entry would continue, and hopefully conclude, my journey towards the perfect Network Attached Storage (NAS) system. The primary reason I want to use one is to stream TV Shows and Movies to my Samsung 40” LCD TV in my Living Room by means of my AppleTV. It also serves as a backup location for my important files, and is now capable of performing server related tasks that were previously handled by the AppleTV. This is my 4th attempt at a NAS solution for the Spitfire Network, the most expensive, the most capable, and finally…the most likely to survive. For a quick NAS summary:
| Eclipse (Gaming PC) | Too loud, power intensive, resource hog no redundancy |  |
| Apple Extreme Wireless Station + USB HDD | Ran drive continuously leading to failure no redundancy |  |
| Linksys NAS 200 | Ridiculously Poor Performance, Noisy |  |
| Atom / FreeNAS | Best of all worlds but expensive, large footprint |  |
I tried all the cheap alternatives to buckling down and forking over the cash for a decent NAS / Server but in the end the only lesson is one we should all be familiar with already - You get what you pay for. In trying to be cheap, all I got in return was lost data (Apple Base Station) and unusable performance (Linksys + hosting files on my gaming pc). Not that a FreeNAS solution has to be expensive however. These days many people have a spare PC in their garage or something. All it needs to be given new life as a FreeNAS server is:
x86 Processor
128 MB ram
32 MB disk space
Network card
The catch is that your hardware has to be supported by FreeBSD, but I’d be willing to bet that if it’s anything similar to a commercial computer, you are all set. I looked into the option of buying a cheap, used PC for under $100 on craigslist but in the end I decided to build a new PC specifically for a few reasons:
1. I knew exactly what hardware it would have, with warranties.
2. By using an Atom chipset it has power consumption comparable to a netbook, which is preferable to a power hungry P4
3. The Atom D510MO chipset is fanless, making the entire computer almost silent
4. Gigabit Ethernet (up from 10/100 on the linksys nas)
I bought a nice looking glossy black case from Newegg, which was running a pretty nice deal on it as long as I signed up for spam, and sold the Linksys NAS for $5 profit. I had the disk drives already from the linksys raid setup, so in the end I just had to buy the case, atom board/cpu combo, and 1 GB of RAM for it. Not too much cost to expand its use greatly, but if you were starting from scratch it might be a big investment. So give it a try with your old garage PC first. I paired the computer with a 23” Samsung 2333HD Display, which is kind of a misnomer. The actual NAS is capable of displaying a typical Unix/bsd console, but it’s really meant to run in a corner or a closet without a keyboard or monitor hooked up. In fact, there is no GUI display included. The monitor is really displaying video from my macbook pro, which makes the computer give the appearance of a hackintosh. It’s also a nice 1080p HDTV. The actual Atom computer runs without being plugged into the monitor, unless I’m debugging something.
So I spent all this money getting decent hardware, at least the software was free! You’ll need to decide between a “full install” or “embedded” install. The full install is like a typical OS installed onto your disk. The embedded install is copied to a USB Flash Drive which is then booted off of, and it runs FreeNAS in a virtual disk created out of RAM. The full install is easier to customize, but the embedded install is blistering fast and allows your hard drives to spin down when idle. I went with the embedded because it’s very important to me for the idle disks to spin down. This saves wear and tear, heat (remember it’s fanless), and power ($). You can’t repartition the boot drive, so I had to add another flash drive to run my custom server apps like rssdler and python. Running these apps off the main data HDDs would prevent them from spinning down (they’d never be idle). Flash drives are cheap SSDs and it works great for NAS purposes.
After the initial console setup, you can unplug the monitor and control the system via a web interface on any other computer on the network. This makes it very easy to choose services, set them up, add disks and shares… You really don’t have to be a pro to get it to work reliable and consistently. I followed a tutorial, added my two hard drives as a redundant software RAID 1 array[Edit 11/30/10: This has changed, see: FreeNAS Update: Rsync and Unison], and added windows and apple file sharing in less than 20 minutes. Performance was a ten-fold increase over my Linksys NAS, averaging about 40MB/s transfers which I’m guessing is limited by gigabit ethernet. But don’t take my word for it, here’s some fancy looking pictures:
|
|
 
|
 
|
My Atom server blows away any of my previous NAS attempts, and I’m finally happy and able to not worry anymore about my hard disks crashing and taking all my stuff with them like the great hard drive crashes of 2003. It’s far quieter and faster than the Linksys NAS, and I can run apps like rssdler off the server alone. This means the AppleTV, which is already slow, doesn’t have to use its few resources on downloading for me anymore. It uses waay less power than my gaming pc did when it stayed on to serve files, and adds the redundancy of two drives, protecting against hardware failure. The loss of one hard drive wouldn’t make me lose any data. The web interface is easy to use, and since it’s a mini-atx form factor, it’s got plenty of room for additional hard drive space so I can expand it over time. The Linksys and many other NAS devices only support a maximum of two disks. And did I mention how quiet it is?
The only disadvantages I could think of was the additional cost for having an additional computer. If you have an old PC and could throw it in the closet or something, this disadvantage may not mean much for you. Also, the mini-atx form factor eats up some desk space for me, but I want to keep it close to my other computers. Again if you threw it in a closet you might not care. I’m decent enough with Linux/Unix/bsd commands that I didn’t have any trouble setting it up, but if you’re a windows lover then buying a commercial NAS might be up your alley.
So in summary, I am totally blown away by FreeNAS and all it’s capable of. I tried everything else, but sometimes when you need the job done right you just have to do it yourself. My next tech post will probably be about AT&T U-Verse, how it’s integrated into my new house, changed my network, and reacted to my experiments.
Check out the free software at: http://freenas.org/freenas
Sunday, December 07, 2008
Slipping the surly bonds of Earth
Oh! I have slipped the surly bonds of Earth
And danced the skies on laughter-silvered wings;
Sunward I’ve climbed and joined the tumbling mirth of sun-split clouds, and done a hundred things
You have not dreamed of wheeled and soared and swung
High in the sunlit silence. Hov’ring there,
I’ve chased the shouting wind along, and flung
My eager craft through footless falls of air…
Up, up the long, delirious, burning blue
I’ve topped the wind-swept heights with easy grace
Where never lark, nor e’er eagle flew-
And, while with silent lifting mind I’ve trod
The high, untrespassed sanctity of space,
Put out my hand and touched the face of God.
John Gillespie McGee
Monday, October 17, 2005
Flight Team results
Back in Spring Semester of 05, myself and the other flight team officers looked to each other and set a goal for our team. We decided that this year, we were going to do well enough at the Regional NIFA Safecon Competition that we would be invited to the national competition. The KSU flight team has not been to the national competition in 19 years. So, let me tell you that this was no small feat; to revamp the team, practice, learn, and keep up with everything else in our lives. A lot of you all know that I haven’t been around a lot for a while. This is why. October 9-16 was the Regional Competition at Western University in Battle Creek Michigan. It was one of the best expiriences of my life. It was cool to see all the other teams and planes and talk to them. It was also really cool to see our team come together from almost nothing and perform so well. Usually 3 teams from our region (about 7 colleges) make it to the national competition. This year they were only going to take the top two. Western Michigan always wins first. Ohio State University always wins second. The third place spot is always fought for. Last year we lost the third spot to Ohio University by 6 points out of hundreds. This year Ohio State University is hosting the National Competition, so it turns out that they go to nationals no matter what. This means that there was still a spot even with W.M. in 1st and OSU in 2nd. Well, this year we worked our asses off all semester and all week. It came down to the awards banquet Saturday night. Our hearts on the line as they announce each of the top 10 winners of each event. When it came down to it, the Kent State University Precision Flight Team….came in second place! We even beat the “unbeatable” Ohio State! That means that the only reason Ohio State will be at nationals is because they are hosting it! It’s so awesome! If you work hard enough for anything you can do it - ANYTHING!