Wednesday, September 26, 2012

CouchDB Clustering and BigCouch

I'm a big fan of CouchDB as a quick and easy document store backend. However, the advertised feature list is sometimes a bit... ambitious. You "can" do a lot of things which you just shouldn't, where a naive application will almost certainly lead to trouble in production.

One such thing is master-master replication. It is easy to set up bi-directional continuous replication between two CouchDB instances. Throw a load balancer in front of them, and in certain, limited situations this could perform as a high-availability cluster.

A serious issue with this setup is handling conflicts when a document is updated on multiple nodes. The CouchDB documentation has a detailed explanation of how it handles conflicts, and suggests some client-side code for getting the right version of a conflicting document.

In practice, I've had a lot of trouble with this scheme in CouchDB 1.1.0. I have seen view indexes fail to update with the merge "winner" on replication, leading to a situation where the view results do not represent the underlying data. And since this "can't happen," there is no easy way to force a reindex.

In any case, it would be much better to have the cluster do the work for us.


The CouchDB Guide has a chapter on clustering, which recommends using the CouchDB Lounge clustering framework. Lounge is a proxy which sits in front of several CouchDB servers. It has a few parts: a "dumb" proxy which redirects non-view requests to any node, a "smart" proxy which fans out views across several nodes, and a replication tool to make data redundant.

I find the guide's recommendation surprising, as there are a few problems with Lounge:

  • The deploy process is very tied to (now-defunct) Meebo's production platform, based on RPMs
  • It hasn't been touched in over three years, despite promises for fixes in "the near future"
  • It relies CouchDB's built-in conflict resolution
  • It relies on a custom patch to CouchDB

Enter BigCouch

Fortunately, the fine people at Cloudant needed CouchDB clusters to actually work in order to make money, so they developed BigCouch to solve these problems.

BigCouch is not without its own issues, but these are mostly political -- since its inception a few years ago there has been talk of merging BigCouch back into CouchDB, yet the passive-aggressive Twitter arguments continue. Also, they do not seem to be particularly worried about build tests.


Setting up a BigCouch cluster on Ubuntu is very easy; I was up and running on precise in minutes. The only hangup was that bigcouch isn't packaged for precise, nor is the version of libicu (exactly 4.4) on which it depends.
  1. Get libicu44
    1. Download the package from Natty Narwhal:
    2. dpkg -i libicu44_4.4.2-2ubuntu0.11.04.1_amd64.deb # or i386
  2. Add cloudant repository for Oneiric Ocelot:
    1. echo "deb oneiric main" | sudo tee /etc/apt/sources.list.d/cloudant.list
    2. apt-get update
  3. apt-get install bigcouch
Oddly, and unlike most services, it starts automatically when you install it. The service is managed by sv (see /etc/services/bigcouch). You can start and stop it with sv up bigcouch and sv down bigcouch. Configuration, libraries, and binaries are installed into /opt/bigcouch.


Now, there is a little configuration (described in more detail in Installing & Using BigCouch).

Edit /opt/bigcouch/etc/vm.args and change:
  • -name
  • -setcookie some_secret_string
etc/defaults.ini and etc/local.ini work much like in CouchDB -- here you can fiddle with ports, enable SSL, etc. Note that the [chttpd] section describes the user-facing CouchDB server, and [httpd] describes the BigCouch "backdoor" used for administration and cluster coordination.

Your nodes will need to talk. Configure your firewall so that they can see each others' ports 5984 (CouchDB), 5986 (BigCouch), and 4369 (Erlang port mapper), plus any used for SSL.

Building the Cluster

To add a node to the cluster, you use the admin JSON API exposed on port 5986:

curl -X PUT -d {}

(And vice versa)

Now, you should be able to interact with the CouchDB server on port 5984 of either node (ignoring completely the BigCouch-ness going on) and changes should appear immediately on both nodes.

The internal sharding and replication mechanism is based on Amazon's Dynamo Paper, which is pretty generally regarded as the definitive guide to robust cluster storage.

The API to BigCouch necessarily differs a bit from CouchDB's: for example, _stats lives on the admin side since it is now cluster-wide. The differences are outlined in the BigCouch API docs.

High Availability, etc.

The nodes talk in the background to ensure consistency and all are peers, so you can connect to any one you want. For high availability, put a load balancer or other such routing in front of the nodes to create a single point of entry. BigCouch recommends HAProxy. (And for an illuminating discussion of why you might use keepalived but not heartbeat, see this post).


    1. Funny you mentioned BigCouch, because back in Janurary they announced they were going to merge BigCouch into CouchDB. Do you know if this merger has completed? The lasted update back in May suggested it is a long way into completion:

    2. Hi, Chris. I'm certainly no insider, but my understanding is that the merger has been about to happen for a really long time. Yet the BigCouch site doesn't mention it, the CouchDB Guide recommends Lounge, and then there is this: So as far as I can tell BigCouch is still the most viable option for CouchDB clustering.

      Hopefully the merger actually happens soon; serious clustering support would a huge win for Couch.