A tiny server saves my internet!

In Hong Kong, we face a unique set of problems. Space is at a premium and our Internet connections are so fast that consumer routers made for mere mortals are too slow for us Hongkongers.
The misleadingly named home "gigabit routers" rarely have the processing power to perform NAT on a single TCP connection at even a few hundred megabits. That's a problem when you live in a city where multiple fiber-to-home internet providers offer gigabit service for around US$30 a month...yes, that includes TV.

It's a very first world problem: most people would be happy to move to Kansas and pay Google more than twice what we Hongkongers pay in order to have the same problem. But as the many threads in local computer forums show, this is no solace when you could be enjoying gigabit but you're not.

The solution is pretty easy - build your own router with a PC and a couple of gigabit network cards.

Imagine my excitement when a HP ProLiant Gen8 MicroServer showed up at my flat right before Typhoon Usagi hit Hong Kong.

The Gen8 MicroServer has 2 gigabit nics (plus a 3rd for HP's management firmware iLO - more on that later) and room for 4 3.5 inch SATA drives. Mine came with the Intel® Pentium® G2020T, 16 gigabytes of ram and a 500 gigabyte hard drive. Plenty of power to route a gigabit with some left over to serve as a home NAS.

The Gen8's Hardware - Perfect for HK

Hong Kong's housing and office prices are the most expensive in the world. You've heard the old adage that time is money. In Hong Kong, space is money. Fixed costs are measured in square feet as much as in dollars. So it's as if someone at HP was thinking specifically about Hong Kong when they designed the Gen8 MicroServer.

The Gen8's best feature is that it packs a lot firepower and flexibility into little more than half a square foot. And it's designed to let you stack multiple MicroServers and their matching network switches. That means a Hong Kong business can have its own in-house data center in the space taken up by the office water cooler.

The Gen8's hardware is designed put a lot of thought into how to maximizing expandability in minimal amount of space. You can tell someone really cared. Drives slide in and out the front without any tools. The case comes apart with two easy on the finger thumb screws. You only need a tool if you want to take the motherboard out, and it's conveniently included inside the front door.

Another plus for Hong Kong's SME and home office market, where people and machines work together in close quarters, is the Gen8's (lack of) noise level. Unlike the Mac Mini Server, which sounds like it's about to take flight under heavy load because of it's tight thermal tolerances, tiny fans and tightly packed interior, the Gen8's well-ventilated interior plus large, rear ventilation fan is quiet and cool no matter how much work you throw at it. The only noise I could hear came from the hard drive and only then if air con was switched off.

Firmware frustration

If only my experience with the Gen8's built in iLO4 management firmware and bios was as good as the hardware. For those that aren't familiar with iLO, it essentially gives you a web browser accessible control panel for your server hardware. It has a cool feature that let's you mount an ISO on a web server as a CD/DVD-ROM drive. Perfect for installing an operating system - especially in Hong Kong with our gigabit internet. As HP says, iLO let's you "Simplify server setup."

The bad news is HP doesn't actually let you use this feature unless you buy an additional license. I understand the business reasoning for doing this, they're trying to segment their customers to charge for the value provided, - but I think this is the wrong way to do it. Why you would want to go out of your way to make sure some customers can't "Simplify server setup" and have a worse user experience is beyond me. That's like taking a perfectly good car, locking the doors and only unlocking them if the customer pays more. "Oh, you don't want to climb through the windows to get into your car? Just buy this additional license and you can use the doors!"

On the upside, HP does have a free 60-day trial license for iLO available. I signed up for the trial license and was able to boot off of the HP-custom ESXi ISO image and install the OS to a USB drive plugged into the internal USB port. Unfortunately, my machine wasn't able to boot off the USB drive after installation, despite the BIOS was set up properly to enable this. I even tried another making another bootable USB drive with a recent Ubuntu version and still wasn't able to get the machine to boot.

No matter, I could still install ESXi directly to the hard drive...except that I couldn't, because I ran into another bug in the Gen8's firmware. iLO had decided that my 60-day trial license had already expired, so it would no longer let me boot off the remote ESXi ISO. The warm, glowing feeling the MicroServer's hardware had left me with was quickly being replaced with the frustration of running into bugs that HP should have caught and fixed during testing. Saving customers from this feeling is exactly why we created Pay4Bugs.

After much messing about, I was finally able to install ESXi through HP's built in "Intelligent Provisioning" feature. Unfortunately, even the "custom install" option wouldn't let me choose where ESXi was installed, so I didn't have the option of trying installation to a different USB drive. It just installed ESXi to the internal hard drive.

Fortunately, once ESXi was up and running I could get back to enjoying the Gen8's beautifully packaged power.

My next step was to install a free Windows 7 virtual machine since VMware doesn't seem to have a Vsphere client for Mac OS X. I later moved this VM to the Gen8 and install VNC, which makes it easy access using Chicken from my Mac both for managing ESXi and using IE10 to test web sites.

I set up two Ubuntu 12.04 LTS virtual machines on the Gen8. One to serve as the router and another as the NAS.

The router

For the router, I assigned eth0 to the vmware bridged network connected to the physical nic port for my local network. Eth1 and eth2 were assigned to virtual nics connected to the second physical pro with two different vlans. I connected each of the two vlans to different ports of my PCCW-provided GPON terminal via the HP PS1810 managed switch that came with my server. Since PCCW provides two DHCP assigned public IPs, this let me assign both to the same router, one to eth1 and one to eth2.

I configured eth1 for general NAT (Masquerading) and set up one-to-one NAT with the second public ip to give one internal machine full, unblocked two-way internet access.

One of the problems of gigabit internet access is finding ways to saturate it. A couple simultaneous downloads of large files hosted on a Softlayer cloud server with a gigabit interface in Singapore, Amazon Cloudfront and Cachefly did the trick. With the Gen8 router set up, I could consistently break 900 megabits. Not too shabby for a $30 / month internet connection.

$ Filezilla

On the NAS virtual machine, I set up ZFS for Linux and created a mirrored ZFS storage pool with one device a virtual disk on the Gen8 and the other a virtual disk shared via iSCSI on another local machine.

root@filezilla:~# zpool status
  pool: tank
 state: ONLINE
  scan: resilvered 35.3G in 0h39m with 0 errors on Fri Sep 27 18:05:17 2013
config:

	NAME                                                        STATE     READ WRITE CKSUM
	tank                                                        ONLINE       0     0     0
	  mirror-0                                                  ONLINE       0     0     0
	    scsi-1494554000000000040e6e84e62a169d163f8a7902e219d51  ONLINE       0     0     0
	    pci-0000:00:10.0-scsi-0:0:1:0                           ONLINE       0     0     0

errors: No known data errors
root@filezilla:~#

I created a ZFS filesystem inside the storage pool for each of my NAS shares. Since I live in a largely Apple world (besides our company's all Ubuntu servers), I went on to set up the open source daemon for Apple's file sharing protocol netatalk. Netatalk used to be a bit temperamental, but now seems to be rock solid. You can even make it look like an Apple Time Capsule by following these instructions. If you come from the Windows world, you'd want to set up samba for file sharing.

Performance isn't the best in the world, but much of that is my fault for making a ZFS mirror with a remote iSCSI virtual drive. It could be easily and cheaply improved by adding an inexpensive, small SSD as a cache in front of the ZFS mirror, essentially letting the MicroServer function like Apple's Fusion drive.

The little server that could

The HP ProLiant MicroServer Gen8 is a well-designed, capable workhorse for small businesses or home offices. The firmware issues I ran into detract from the "it just works" hardware design, which is unfortunate, yet something that HP could easily fix in a future update.

While it let me speed up my internet and store some extra data, what you could do with it in Hong Kong with inexpensive gigabit internet is so much more exciting that what I've done. For example, imagine you were a small video production company. You could fill the Gen8's four drive bays with SSDs. The SSDs would be your ZFS cache and your ZFS storage devices would be remote iSCSI targets. It could serve as your office's gateway to unlimited remote NAS storage with onsite NAS performance.

What would you do with yours?