Thursday, January 11, 2018

The Light Box: A DIY STEM Toy

My niece, age two, is obsessed with lights. Entering a room, she will point at each fixture saying "light! light! light!" until you turn them all on. I think she keeps a complete mental inventory of every light-producing object she has encountered. She learned very early on that she could control the lights using a wall switch. This got me thinking: what kind of action-at-a-distance magic must this look like to a two-year-old? A lot of things must seem like magic!

I am an experimental particle physicist, so trying to make sense of the natural world is sort of my job! In that spirit, I set out to build a toy for my niece which would build upon her natural curiosity and ideally help teach something about the universe.

💡 "The Light Thing"

The idea is this: a set of light switches controls a set of lights, but the connection between them is through a set of patch cables which can be rearranged like a switchboard. The goal is to emphasize the physical connection between the switch and lightbulb, and provide a simple playground for experimenting with circuits.

A few design constraints: it has to be super safe, it has to be uncomplicated, and the switches and bulbs have to be the real thing.

I imagined a sort of console, which would sit on the ground or a table:

Concept art

Construction Phase

Fortunate indeed that the audience is a two year old... I am no carpenter. But really, a drill, screwdriver, small hand saw, a keyhole saw, and something to measure with are the only essential tools.

The first step was laying out some parts to get a sense for what size would really work. It had to fit three ceramic sockets across, at least, but how much room for the patch cables?

Trying it on for size...

Somewhat arbitrarily, the footprint is 21" x 17" and it tapers from 8" down to 2.5" in height.

The box consists of a wood frame built from 3/4" square dowels cladded with 1/4" oak plywood. 

The frame

With the frame ready to go, it's just a matter of cutting the plywood to size, cutting rectangular holes for the light switches with a keyhole saw, and drilling holes for the light switch wires (under the ceramic sockets) and the 1/4" jacks for the "switchboard."

A partial assembly

The Lights

Perhaps the most straightforward way to do this would be to actually power the lights on 120V AC coming in the back, and the switches and patch cables just carry a small signaling voltage to a relay that actually turns the light on.

However, in order to be super safe, and also to make it portable, I decided to make the whole thing low voltage. But still the lightbulbs had to look like real-world/grown-up lightbulbs. So I ripped apart the lightbulbs and did my own thing inside. With my Sylvania #74311 bulbs, it turned out to be possible, if tricky, to do this without completely destroying them.

The original insides

I carefully pried off the dome and removed the board, yanking out the HV board behind it in the neck of the bulb. The ground and hot wires weren't soldered in, they were just held in place by pressure -- by the screw threads on the body for the ground, and the electrical foot contact which is just a pin that pops out. In 1 of 3 cases, the ground wire didn't pop out and I didn't have to remove the screw threads, which really made things easier.

Reusing the original board as a structural base, I replaced the whole insides with a set of three bright white LEDs in parallel plus a 100 Ohm resistor, on a little perf board raft. This part could certainly be improved a lot, in particular to make the illumination more uniform, but gets it close enough.

The new insides

At this point I also sanded off the regulatory markings, which certainly no longer apply.


With the modified LED bulbs in hand, the rest of the wiring was straightforward. After staining the cut plywood a nice dark walnut and putting on a few coats of Polycrylic, I wired up and installed the sockets and switches, and installed six 1/4" mono jacks (as for an electric guitar, but long-shafted ones that can make it through the 1/4" plywood).

Front face and inside wiring

Here is a schematic illustrating the wiring. Not much to it! (For tips on reading schematic symbols, check out here!)

Wiring schematic

The inside view shows the positioning of the four D-cell battery pack at rear of the cabinet. For access, there is a little door with a latch:

Battery access hatch

Finished Product

Here are a few shots of the finished product:

And the most important part... a satisfied customer!


Bill of Materials

Here's a list of parts I used:

Tuesday, January 5, 2016

Google Fi and the Nexus 6P for Humans

When it came time to replace my ailing phone, a Verizon Galaxy Nexus, I thought I'd give Google's fledgling phone service Project Fi a shot, along with the new Nexus 6P.

There are a lot of reviews of 6P, mostly positive but occasionally scary. There are fewer reviews for Fi, and they're pretty mixed. Now, I'm not a tech blogger juggling all the season's hottest phones: like most, I have one and it needs to work. And since it can be hard to sort out the good and the bad, I thought I'd share my experience with Project Fi and the Nexus 6P, from a user's perspective.


  • Spiffy Nexus phone: The Nexus 6P is mostly best-in-class, and relatively inexpensive for a flagship phone. It is also among the first with Android Marshmallow. 
  • Project Fi potentially awesome: Transparent switching from wifi to cellular calling. Using public wifi with automatic VPN through Google. This sounds like the future of mobile data and it's cool to get a preview.
  • Nothing to lose: No contract lock-in
  • Size: The Nexus 6P is enormous compared to my current Galaxy Nexus. I made a cardboard mockup to see how it felt. It's pretty big.

  • Quality?: There are some really scary reports out there about the quality of the Nexus 6P. They bend. The glass on the back shatters. The front glass shatters. Basically I am willing to accept that this is either bullshit or Google/Huawei will fix it. I mean, I bet I could bend my laptop, too... but I won't? As an aside, it pains me greatly to see these phone bending videos while I wait for weeks for my device.
  • Coverage?: Reviews of the Project Fi service are mixed. Gizmodo reviewers called it a disaster but soon decided that it was instead great and they would probably replace their current service. A lot of reviews say it's a great "second" phone service. Who does that!?
The bottom line is that for people who spend a lot of time on wifi/don't use a lot a lot of mobile data, a 5X/6P on Project Fi provides a cheaper alternative to other carriers. High-end phones are mostly in the $600 range, and typically not subsidized like they used to be. Other talk and data plans cost about the same as Fi, but often have an additional ~$20 charge per line, so you'd be paying $50/month for talk + 1 GB, as opposed to $30/month on Project Fi. Do the math based on your typical usage, which you can find in the settings of an existing phone.

Sign-up and Ordering

I headed over to and asked for an invite. Six days later, I was in. Signing up was really, really easy. There were a few basic questions to the tune of:
  • Are you bringing your own phone or buying one? Project Fi only works with the Nexus 6 and the new Nexus 5X and Nexus 6P. If you buy a new phone, there is an option to buy it up front or finance it interest-free over two years.
  • How much data? You have to pick an integer number of GB, but they will reimburse/charge you if you go under/over. As far as I can tell this exists just as a guide to help you budget.
  • Connect to Google Voice? If you have Google Voice, Fi can eat your account. Or you can transfer a number from another carrier.
They've done a very nice job with the user experience. I clicked through some pretty pages and in 10 minutes I had a new service and a phone on the way. Even if the service doesn't pan out, I'd like to believe this was a glimpse at the future of phone/plan shopping, at least.

When the order is submitted, nothing changes. If applicable, a credit card isn't charged until the device ships, and they don't touch your Gvoice account until you activate. I got a friendly order confirmation, access to the Fi account website (currently rather boring), and a 50% off coupon for a phone case (although they seem to always be sold out of the best one).

At the time of order, my estimated lead time for the 32GB Graphite Nexus 6P was five weeks; ordered in mid-November and expected mid-December.


Ahead of schedule on Monday, December 1, I got word that my phone and Project Fi SIM were en route via FedEx ground. I quickly ran over to Amazon to pick up the well-reviewed Spigen 6P case since the nice-looking fuzzy Nexus 6P case on the Google Store was still out of stock. It all showed up that Friday:

Phone in a box, SIM in a folder.

I am an artist.

The setup is the usual business. Click through a few things, it updates itself, you decide whether to keep your Google Voice number of not. In my case, it defaulted to no, and if I hadn't changed it I would have lost my number. Beware!

At some point, you can transfer settings from another device, which worked well. I opened Google Settings on my old Galaxy Nexus, picked the 6P, and magic ensued. Amusingly, the My Verizon app showed up on the 6P. I appreciated that since I finally know the satisfaction of deleting it.

You have to activate the Fi service, and it warned me that this could take a day or so, during which the device couldn't make or take calls. I went for it, and it actually took less than five minutes. Hooray!

First Impressions

In short, it's pretty great. There are tons of details and specs in proper reviews, so this is more about feelings.

  • Performance: Nothing to say. It's so fast it doesn't matter.
  • Camera: Quite good. Quick focus, well-balanced colors, and exceptional low-light performance. I can actually take one picture and it's good, as opposed to the five I needed to take on the Gnex to get something halfway decent. Side-by-side, I think it's a bit better than the Droid Turbo, despite having lower resolution; certainly less noise in the shadows.
  • OS: A major selling point for me was the pure Android OS, and it doesn't disappoint. Thoughtfully designed everywhere, Marshmallow is perfectly matched to this phone. I find Google Now On Tap is hit-or-miss: sometimes super-handy, sometimes has nothing to say. Hopefully they continue to improve it.
  • Fingerprint Scanner: Fingerprint unlocking is super cool. The scanner is in a natural position for me and it takes about as long to work as it does to lift up the phone, so it's quite seamless.
  • Battery: With reasonable use, the 6P lasts me over two days. I charge it every other night.
  • A/V: The display is crisp, well-saturated, and huge. There are speakers at the top and bottom, for stereo sound in landscape orientation. They're pretty loud, and pretty good for a phone.
Camera: Cat helping with dissertation.

Camera: I can see the pixels on that science!

Camera: Low light

Project Fi:
  • Project Fi App: The app is well-designed and intuitive, and the whole experience feels way more simple and transparent than dealing with traditional carriers.
  • Phone Calls: I called on WiFi. I called not on WiFi. It worked fine.
  • LTE: Mobile data around town? Great. Drive from Philadelphia to Bethlehem, PA rocking out to some Google Music? Also fine.
  • WiFi: At a coffee shop, I automatically connected to the public WiFi and a little key showed up, presumably to indicate that I was using Google's VPN as expected. Cool.
  • Nexus Protect: Recently Google launched Nexus Protect, a repair/replacement plan, which adds $5/month to the Fi service. They'll replace the device twice, fix broken screens, etc. I went for it.
I was prepared for a disaster: dropped calls, no service outside major cities, constantly switching networks and losing connection. Nope. It's all good. Driving through campus would destroy my Galaxy Nexus as it wavered between 4G and WiFi. Not so with the 6P, which seems a lot more intelligent about deciding which networks to use/avoid. The only time I had a poor signal was driving through the mountains of West Virginia on I-70 (where a Verizon phone did indeed had much better service). Elsewhere, including outside cities, it was fine and typically 4G.

Ever the thoughtful wireless provider, Google/Project Fi sent me a Christmas present, a set of legos with instructions to build a phone holder or cable organizer, and an extra USB cable.

Verizon never sent me toys...


So far so good! About a month in and the phone hasn't shattered into a thousand pieces yet, and Project Fi really feels like the carrier of the future, not a beta test.

So far, I have experienced none of the issues with hardware build quality, software problems, service quality, etc., that I read about in reviews and blogs. Hopefully it stays that way, but I'll update this if I run into trouble!

I'd recommend both the 6P and Project Fi to anyone. Indeed it seems especially well-suited to the not-so-tech savvy crowd: it's simpler to understand what you're paying for than with most traditional carriers (even when they're not cheating you) and there is in-person 24/7 support.

Friday, April 11, 2014

HighPoint RocketRAID 2320 (rr232x) on Ubuntu 12.04

HighPoint was not very forward-thinking when they produced the RocketRAID 2320, and the only available drivers assume that you're running 2.4 <= kernel <= 2.6.  Like, really badly:

$ uname -r
$ make
../../../inc/linux/Makefile.def:85: *** Only kernel 2.4/2.6 is supported but you use 2.2.  Stop.

Oof. They also don't compile when you remove that check, because kernel headers and a the prototypes for a handful of system calls have changed. Incredibly, this device is still on sale despite the obvious lack of interest from the manufacturer -- if you're looking for a cheap RAID controller for Linux, look elsewhere!

The latest rr232x driver version (1.10) doesn't appear to be on the HighPoint site, but fortunately there is a mirror here: [blog post].

Untar that, then grab and gunzip this patch: [blog post].

Apply the patch:

$ cd rr232x-linux-src-v1.10
$ patch -p1 < /path/to/rr232x-linux-src-v1.10.patch

Then build:

$ cd product/rr232x/linux
$ make
$ sudo make install
$ sudo modprobe rr232x  # load it up!

You should see the device initialization (and any drives enumerated) in dmesg.

Thursday, September 27, 2012

Evil C++ 6: Default Method Return Values!?

Caveat: Contents certainly platform-dependent -- these results are from GCC 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3). Your mileage may vary. If you get different results on other platforms, I'd love to hear about it in the comments!

Today I came across a wonderfully devious C++ gotcha, which tops my "evil C++" charts so far: failing to return x at the end of a non-void member function, you get as the return value the address of the instance (i.e. this), cast to the return type of the function.

Of course, one should always return something from a non-void function. But it's an easy mistake to make, due to typo or misconception.

Consider, for example:

Quite surprisingly, this compiles with no errors or warnings by default!

Enabling -Wreturn-type (which comes with -Wall) does get us this:

$ g++ -o null_member null_member.cpp -g -O0 -Wall
null_member.cpp: In member function ‘int foo::thinger()’:
null_member.cpp:5:24: warning: no return statement in function returning non-void [-Wreturn-type]

It seems like this should always be an error... there is never a time that this is a good idea.

In any case, the program outputs:

$ ./null_member 
f @ 0x7fffc30e54cf
f.thinger() = c30e54cf

Clearly, nothing in the C++ code is doing this.

But having studied Apple IIe assembly for half a semester back in high school, I thought I'd try my luck with the GDB disassembler.

$ gdb ./null_member
Reading symbols from null_member...done.
(gdb) disassemble main
Dump of assembler code for function main(int, char**):
   0x00000000004004f4 <+0>:     push   %rbp
   0x00000000004004f5 <+1>:     mov    %rsp,%rbp
   0x00000000004004f8 <+4>:     sub    $0x20,%rsp
   0x00000000004004fc <+8>:     mov    %edi,-0x14(%rbp)
   0x00000000004004ff <+11>:    mov    %rsi,-0x20(%rbp)
   0x0000000000400503 <+15>:    lea    -0x1(%rbp),%rax
   0x0000000000400507 <+19>:    mov    %rax,%rdi
   0x000000000040050a <+22>:    callq  0x400544 <foo::thinger()>
   0x000000000040050f <+27>:    mov    %eax,-0x8(%rbp)
   0x0000000000400512 <+30>:    lea    -0x1(%rbp),%rax
   0x0000000000400516 <+34>:    mov    %rax,%rsi
   0x0000000000400519 <+37>:    mov    $0x40063c,%edi
   0x000000000040051e <+42>:    mov    $0x0,%eax
   0x0000000000400523 <+47>:    callq  0x4003f0 <printf@plt>
   0x0000000000400528 <+52>:    mov    -0x8(%rbp),%eax
   0x000000000040052b <+55>:    mov    %eax,%esi
   0x000000000040052d <+57>:    mov    $0x400644,%edi
   0x0000000000400532 <+62>:    mov    $0x0,%eax
   0x0000000000400537 <+67>:    callq  0x4003f0 <printf@plt>
   0x000000000040053c <+72>:    mov    $0x0,%eax
   0x0000000000400541 <+77>:    leaveq 
   0x0000000000400542 <+78>:    retq   
End of assembler dump
(gdb) disassemble foo::thinger
Dump of assembler code for function foo::thinger():
   0x0000000000400544 <+0>:     push   %rbp
   0x0000000000400545 <+1>:     mov    %rsp,%rbp
   0x0000000000400548 <+4>:     mov    %rdi,-0x8(%rbp)
   0x000000000040054c <+8>:     pop    %rbp
   0x000000000040054d <+9>:     retq   
End of assembler dump.

Compare the last bit to the assembly for a thinger that returns 42:

Dump of assembler code for function foo::thinger():
   0x0000000000400544 <+0>:     push   %rbp
   0x0000000000400545 <+1>:     mov    %rsp,%rbp
   0x0000000000400548 <+4>:     mov    %rdi,-0x8(%rbp)
   0x000000000040054c <+8>:     mov    $0x2a,%eax
   0x0000000000400551 <+13>:    pop    %rbp
   0x0000000000400552 <+14>:    retq   
End of assembler dump.

In main(), at instruction 0x40050f, register %eax is copied into -0x8(%rbp), or the location of variable i; the 32-bit accumulator register %eax is used to store the return value. Presumably this is a shortcut; writing to a known register is quicker than pushing a return value onto the stack.

In the latter thinger, 0x2a (42) is written to %eax at instruction 0x40054c, but in the former, we don't do anything. %eax is just whatever it happened to be... essentially an uninitialized variable.

In the case of normal (static) function call, this is true -- %eax is just some leftover junk. But for a method call, GCC generates the following:

   0x00000000004004d2 <+30>:    lea    -0x1(%rbp),%rax
   0x00000000004004d6 <+34>:    mov    %rax,%rdi
   0x00000000004004d9 <+37>:    callq  0x4004e6 <foo::thinger()>
   0x00000000004004de <+42>:    mov    %eax,-0x8(%rbp)

lea (load equivalent address) copies the address of f (here, one byte before the base pointer %rbp) into %rax for temporary storage. Since %rax is a 64-bit wide register of which %eax comprises the lower half, %eax is left with part of the address of f, and that's what gets interpreted as the return value. The call to foo::thinger was expected to modify it, but didn't.

It's a wonderful piece of evil. It's a plausible typo bug which compiles without error and causes functions to return corrupt data. It depends on compiler- and machine-specific, assembly-level implementation details, invisible in the source code. Bug reports will vary by platform. And I have no evidence of this, but "no return means return NULL-ish" sure sounds like common C++ misconception.

I caught it because I happened to be returning a pointer, which caused a segfault on deference. But a float could easily go unnoticed, as could a pointer of the same class as self.

Happy coding, and beware of C++!

Wednesday, September 26, 2012

CouchDB Clustering and BigCouch

I'm a big fan of CouchDB as a quick and easy document store backend. However, the advertised feature list is sometimes a bit... ambitious. You "can" do a lot of things which you just shouldn't, where a naive application will almost certainly lead to trouble in production.

One such thing is master-master replication. It is easy to set up bi-directional continuous replication between two CouchDB instances. Throw a load balancer in front of them, and in certain, limited situations this could perform as a high-availability cluster.

A serious issue with this setup is handling conflicts when a document is updated on multiple nodes. The CouchDB documentation has a detailed explanation of how it handles conflicts, and suggests some client-side code for getting the right version of a conflicting document.

In practice, I've had a lot of trouble with this scheme in CouchDB 1.1.0. I have seen view indexes fail to update with the merge "winner" on replication, leading to a situation where the view results do not represent the underlying data. And since this "can't happen," there is no easy way to force a reindex.

In any case, it would be much better to have the cluster do the work for us.


The CouchDB Guide has a chapter on clustering, which recommends using the CouchDB Lounge clustering framework. Lounge is a proxy which sits in front of several CouchDB servers. It has a few parts: a "dumb" proxy which redirects non-view requests to any node, a "smart" proxy which fans out views across several nodes, and a replication tool to make data redundant.

I find the guide's recommendation surprising, as there are a few problems with Lounge:

  • The deploy process is very tied to (now-defunct) Meebo's production platform, based on RPMs
  • It hasn't been touched in over three years, despite promises for fixes in "the near future"
  • It relies CouchDB's built-in conflict resolution
  • It relies on a custom patch to CouchDB

Enter BigCouch

Fortunately, the fine people at Cloudant needed CouchDB clusters to actually work in order to make money, so they developed BigCouch to solve these problems.

BigCouch is not without its own issues, but these are mostly political -- since its inception a few years ago there has been talk of merging BigCouch back into CouchDB, yet the passive-aggressive Twitter arguments continue. Also, they do not seem to be particularly worried about build tests.


Setting up a BigCouch cluster on Ubuntu is very easy; I was up and running on precise in minutes. The only hangup was that bigcouch isn't packaged for precise, nor is the version of libicu (exactly 4.4) on which it depends.
  1. Get libicu44
    1. Download the package from Natty Narwhal:
    2. dpkg -i libicu44_4.4.2-2ubuntu0.11.04.1_amd64.deb # or i386
  2. Add cloudant repository for Oneiric Ocelot:
    1. echo "deb oneiric main" | sudo tee /etc/apt/sources.list.d/cloudant.list
    2. apt-get update
  3. apt-get install bigcouch
Oddly, and unlike most services, it starts automatically when you install it. The service is managed by sv (see /etc/services/bigcouch). You can start and stop it with sv up bigcouch and sv down bigcouch. Configuration, libraries, and binaries are installed into /opt/bigcouch.


Now, there is a little configuration (described in more detail in Installing & Using BigCouch).

Edit /opt/bigcouch/etc/vm.args and change:
  • -name
  • -setcookie some_secret_string
etc/defaults.ini and etc/local.ini work much like in CouchDB -- here you can fiddle with ports, enable SSL, etc. Note that the [chttpd] section describes the user-facing CouchDB server, and [httpd] describes the BigCouch "backdoor" used for administration and cluster coordination.

Your nodes will need to talk. Configure your firewall so that they can see each others' ports 5984 (CouchDB), 5986 (BigCouch), and 4369 (Erlang port mapper), plus any used for SSL.

Building the Cluster

To add a node to the cluster, you use the admin JSON API exposed on port 5986:

curl -X PUT -d {}

(And vice versa)

Now, you should be able to interact with the CouchDB server on port 5984 of either node (ignoring completely the BigCouch-ness going on) and changes should appear immediately on both nodes.

The internal sharding and replication mechanism is based on Amazon's Dynamo Paper, which is pretty generally regarded as the definitive guide to robust cluster storage.

The API to BigCouch necessarily differs a bit from CouchDB's: for example, _stats lives on the admin side since it is now cluster-wide. The differences are outlined in the BigCouch API docs.

High Availability, etc.

The nodes talk in the background to ensure consistency and all are peers, so you can connect to any one you want. For high availability, put a load balancer or other such routing in front of the nodes to create a single point of entry. BigCouch recommends HAProxy. (And for an illuminating discussion of why you might use keepalived but not heartbeat, see this post).

    Thursday, May 10, 2012

    Creating custom directives in a Python docutils ReST parser

    Parsing some ReStructured Text with Python's docutils, and want to make your own custom directives? Here's a minimal example.

    Tuesday, April 24, 2012

    Intel RS2WC080 + 3TB Hitachi DeskStar + Ubuntu 10.04 LTS

    Ubuntu 10.04 LTS is good for servers (long lifecycle), but the packaged drivers for LSI MegaRAID may be too old for the latest PCI-E LSI RAID chipsets (SAS200x, etc.). Mostly these chipsets are found in other vendors' products, and driver source availability varies.

    Test system:

    • Intel DX68SO2 Motherboard
    • Intel Xeon W3565 CPU
    • Intel RS2WC080 RAID Controller
    • 1x 32 GB OCZ Onyx SSD
    • 12x 3 TB Hitachi Deskstar H3IK30003254SW
    • Ubuntu 10.04.3 LTS
    where the OS lives on the SSD and the 12x comprise a (software) RAID 5 array. Eight Deskstars are on the Intel RAID controller; the remaining four are on the motherboard SATA controller.

    The RAID controller found the eight attached drives and automatically configured them as individual JBODs. dmesg shows that the controller initialized properly (module megaraid_sas v4.7) and lists the drives, but they are not enumerated.

    Since Intel only provided RPMs, I went to LSI's site to grab the latest driver source. They don't explicitly provide downloads for "Intel" products, but it's all the same driver, so I found a similar card (this one) and grabbed the zip file for Ubuntu 10.04 LTS (v5.30 at time of writing).

    The zip file contains three tarballs (whatever!): compiled modules for two specific kernels, and the source. Following the included instructions for "recompiling,"
    1. apt-get install build-essential libncurses5 libncurses5-dev linux-headers-`uname -r`
    2. ln -s /usr/src/linux-headers-`uname -r` /usr/src/linux
    3. unzip
    4. tar zxvf megaraid_sas-v00.00.05.30-src.gz; cd megaraid_sas-v00.00.05.30
    5. make -C /lib/modules/`uname -r`/build/ M=`pwd`
    6. cp megaraid_sas.ko /lib/modules/`uname -r`/kernel/drivers/scsi/megaraid
    7. mv /boot/initrd.img-`uname -r` ~/initrd.img.backup
    8. update-initramfs -c -k `uname -r`
    Note that step 6 is wrong in LSI's instructions -- the driver lives in scsi/megaraid/, not in scsi/.

    You should now be able to install the new module (no rebooting):
    1. rmmod megaraid_sas
    2. cd /lib/modules/`uname -r`/kernel/drivers/scsi/megaraid
    3. insmod megaraid_sas.ko
    `modinfo megaraid_sas` should report the correct version number (5.30), and dmesg should show the drives enumerated.

    Now, RAID 'em up! For example (assuming drives have been partitioned), mdadm -Cv /dev/md0 -l5 -n12 /dev/sd[bcdefghijklm]1. Party on.