The cost of storage arrays is falling along with the quality of their manufacture. To quote Lev Andropov from the “Armageddon”: “Components. American components, Russian Components, ALL MADE IN TAIWAN!” What in the nineties used to be quality storage hardware from Sun, these days is a bunch of consumer-quality hard drives thrown together into a poorly designed array enclosure.

For certain super-critical applications customers are expecting absolute data integrity and around-the-clock availability of their new pricey storage hardware. One way to solve the issue of low hardware quality and lousy vendor support is to throw money at the problem and to design a highly redundant storage system. Such a design would waste lots of space but it would ensure that data is accessible no matter what components fail.

RAID-5 is good but in the past I’ve lost three drives in a single RAID-5 set on a Sun 3510 in a matter of hours, leading to downtime and data loss. It turned out that there was a defective batch of drives. In a situation like this, when one drive goes, the RAID set starts rebuilding to a spare drive, generating lots of disk activity and hitting those bad blocks on the defective drive. It’s like a chain reaction. HA clusters – either Sun or VCS – are very impressive, but they are convoluted, expensive and, when they do fail, they are difficult to recover.

A Case Study

I had a pair of Sun 6140s and a pair of Sun v1280s – one production and one stand-by. I needed a design that ensured data and application availability regardless of what hardware component decided to fail: a disk, a bunch of disks, a NIC, both FC cards, a fiber cable, a tray, an entire array, or a server. Each array provided far more disk space than I needed, so I was in a position to waste disk space in exchange for redundancy. I never liked Sun HA Cluster or VCS – too complicated, too unreliable – failing over for a reason or without one – and requiring too much time to recover when the HA software screws up. Here’s a design I came up with:

To summarize: each v1280 has two dual-channel FCs, each 6140 has two quad-channel controllers. The servers are connected to both arrays in a way that would ensure an alternative communication path regardless of a failed device or component. This is how the physical connections are arranged on the outside. As for internal configuration:

Each array has 16 disks. These are arranged in 7 LUNs, 2 disks each with two disks left as spares for the hardware RAID-5. One LUN in each array is reserved as hot spare for the volume manager to use. Each logical volume is based on two mirrored (via the volume manager) LUNs – one from each array. Essentially, arrays are mirror copies of each other.

Additionally, each volume is replicated twice on each server and mounted under different paths. Rsync is used to synchronize the volumes on a regular basis. Not the most advanced solution, considering that Veritas VM is used as well as hardware RAID, but rsync mirroring is the last line of defense, when everything else fails.

So what you get in the end is 1/6 of usable disk space. Wasteful? Very. But when there is ample funding and extreme availability requirements, a configuration like this may be the ticket. I had this system implemented in a production environment. Since then we lost hard drives, FC cards (those go on a regular basis), and once an entire disk array went under, when the primary power supply failed and the backup power supply did not take over because of some firmware problem. Still, there was no downtime and no data loss.