Thursday, September 27, 2012

Evil C++ 6: Default Method Return Values!?

Caveat: Contents certainly platform-dependent -- these results are from GCC 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3). Your mileage may vary. If you get different results on other platforms, I'd love to hear about it in the comments!

Today I came across a wonderfully devious C++ gotcha, which tops my "evil C++" charts so far: failing to return x at the end of a non-void member function, you get as the return value the address of the instance (i.e. this), cast to the return type of the function.

Of course, one should always return something from a non-void function. But it's an easy mistake to make, due to typo or misconception.

Consider, for example:

Quite surprisingly, this compiles with no errors or warnings by default!

Enabling -Wreturn-type (which comes with -Wall) does get us this:

$ g++ -o null_member null_member.cpp -g -O0 -Wall
null_member.cpp: In member function ‘int foo::thinger()’:
null_member.cpp:5:24: warning: no return statement in function returning non-void [-Wreturn-type]

It seems like this should always be an error... there is never a time that this is a good idea.

In any case, the program outputs:

$ ./null_member 
f @ 0x7fffc30e54cf
f.thinger() = c30e54cf

Clearly, nothing in the C++ code is doing this.

But having studied Apple IIe assembly for half a semester back in high school, I thought I'd try my luck with the GDB disassembler.

$ gdb ./null_member
Reading symbols from null_member...done.
(gdb) disassemble main
Dump of assembler code for function main(int, char**):
   0x00000000004004f4 <+0>:     push   %rbp
   0x00000000004004f5 <+1>:     mov    %rsp,%rbp
   0x00000000004004f8 <+4>:     sub    $0x20,%rsp
   0x00000000004004fc <+8>:     mov    %edi,-0x14(%rbp)
   0x00000000004004ff <+11>:    mov    %rsi,-0x20(%rbp)
   0x0000000000400503 <+15>:    lea    -0x1(%rbp),%rax
   0x0000000000400507 <+19>:    mov    %rax,%rdi
   0x000000000040050a <+22>:    callq  0x400544 <foo::thinger()>
   0x000000000040050f <+27>:    mov    %eax,-0x8(%rbp)
   0x0000000000400512 <+30>:    lea    -0x1(%rbp),%rax
   0x0000000000400516 <+34>:    mov    %rax,%rsi
   0x0000000000400519 <+37>:    mov    $0x40063c,%edi
   0x000000000040051e <+42>:    mov    $0x0,%eax
   0x0000000000400523 <+47>:    callq  0x4003f0 <printf@plt>
   0x0000000000400528 <+52>:    mov    -0x8(%rbp),%eax
   0x000000000040052b <+55>:    mov    %eax,%esi
   0x000000000040052d <+57>:    mov    $0x400644,%edi
   0x0000000000400532 <+62>:    mov    $0x0,%eax
   0x0000000000400537 <+67>:    callq  0x4003f0 <printf@plt>
   0x000000000040053c <+72>:    mov    $0x0,%eax
   0x0000000000400541 <+77>:    leaveq 
   0x0000000000400542 <+78>:    retq   
End of assembler dump
(gdb) disassemble foo::thinger
Dump of assembler code for function foo::thinger():
   0x0000000000400544 <+0>:     push   %rbp
   0x0000000000400545 <+1>:     mov    %rsp,%rbp
   0x0000000000400548 <+4>:     mov    %rdi,-0x8(%rbp)
   0x000000000040054c <+8>:     pop    %rbp
   0x000000000040054d <+9>:     retq   
End of assembler dump.

Compare the last bit to the assembly for a thinger that returns 42:

Dump of assembler code for function foo::thinger():
   0x0000000000400544 <+0>:     push   %rbp
   0x0000000000400545 <+1>:     mov    %rsp,%rbp
   0x0000000000400548 <+4>:     mov    %rdi,-0x8(%rbp)
   0x000000000040054c <+8>:     mov    $0x2a,%eax
   0x0000000000400551 <+13>:    pop    %rbp
   0x0000000000400552 <+14>:    retq   
End of assembler dump.

In main(), at instruction 0x40050f, register %eax is copied into -0x8(%rbp), or the location of variable i; the 32-bit accumulator register %eax is used to store the return value. Presumably this is a shortcut; writing to a known register is quicker than pushing a return value onto the stack.

In the latter thinger, 0x2a (42) is written to %eax at instruction 0x40054c, but in the former, we don't do anything. %eax is just whatever it happened to be... essentially an uninitialized variable.

In the case of normal (static) function call, this is true -- %eax is just some leftover junk. But for a method call, GCC generates the following:

   0x00000000004004d2 <+30>:    lea    -0x1(%rbp),%rax
   0x00000000004004d6 <+34>:    mov    %rax,%rdi
   0x00000000004004d9 <+37>:    callq  0x4004e6 <foo::thinger()>
   0x00000000004004de <+42>:    mov    %eax,-0x8(%rbp)

lea (load equivalent address) copies the address of f (here, one byte before the base pointer %rbp) into %rax for temporary storage. Since %rax is a 64-bit wide register of which %eax comprises the lower half, %eax is left with part of the address of f, and that's what gets interpreted as the return value. The call to foo::thinger was expected to modify it, but didn't.

It's a wonderful piece of evil. It's a plausible typo bug which compiles without error and causes functions to return corrupt data. It depends on compiler- and machine-specific, assembly-level implementation details, invisible in the source code. Bug reports will vary by platform. And I have no evidence of this, but "no return means return NULL-ish" sure sounds like common C++ misconception.

I caught it because I happened to be returning a pointer, which caused a segfault on deference. But a float could easily go unnoticed, as could a pointer of the same class as self.

Happy coding, and beware of C++!

Wednesday, September 26, 2012

CouchDB Clustering and BigCouch

I'm a big fan of CouchDB as a quick and easy document store backend. However, the advertised feature list is sometimes a bit... ambitious. You "can" do a lot of things which you just shouldn't, where a naive application will almost certainly lead to trouble in production.

One such thing is master-master replication. It is easy to set up bi-directional continuous replication between two CouchDB instances. Throw a load balancer in front of them, and in certain, limited situations this could perform as a high-availability cluster.

A serious issue with this setup is handling conflicts when a document is updated on multiple nodes. The CouchDB documentation has a detailed explanation of how it handles conflicts, and suggests some client-side code for getting the right version of a conflicting document.

In practice, I've had a lot of trouble with this scheme in CouchDB 1.1.0. I have seen view indexes fail to update with the merge "winner" on replication, leading to a situation where the view results do not represent the underlying data. And since this "can't happen," there is no easy way to force a reindex.

In any case, it would be much better to have the cluster do the work for us.


The CouchDB Guide has a chapter on clustering, which recommends using the CouchDB Lounge clustering framework. Lounge is a proxy which sits in front of several CouchDB servers. It has a few parts: a "dumb" proxy which redirects non-view requests to any node, a "smart" proxy which fans out views across several nodes, and a replication tool to make data redundant.

I find the guide's recommendation surprising, as there are a few problems with Lounge:

  • The deploy process is very tied to (now-defunct) Meebo's production platform, based on RPMs
  • It hasn't been touched in over three years, despite promises for fixes in "the near future"
  • It relies CouchDB's built-in conflict resolution
  • It relies on a custom patch to CouchDB

Enter BigCouch

Fortunately, the fine people at Cloudant needed CouchDB clusters to actually work in order to make money, so they developed BigCouch to solve these problems.

BigCouch is not without its own issues, but these are mostly political -- since its inception a few years ago there has been talk of merging BigCouch back into CouchDB, yet the passive-aggressive Twitter arguments continue. Also, they do not seem to be particularly worried about build tests.


Setting up a BigCouch cluster on Ubuntu is very easy; I was up and running on precise in minutes. The only hangup was that bigcouch isn't packaged for precise, nor is the version of libicu (exactly 4.4) on which it depends.
  1. Get libicu44
    1. Download the package from Natty Narwhal:
    2. dpkg -i libicu44_4.4.2-2ubuntu0.11.04.1_amd64.deb # or i386
  2. Add cloudant repository for Oneiric Ocelot:
    1. echo "deb oneiric main" | sudo tee /etc/apt/sources.list.d/cloudant.list
    2. apt-get update
  3. apt-get install bigcouch
Oddly, and unlike most services, it starts automatically when you install it. The service is managed by sv (see /etc/services/bigcouch). You can start and stop it with sv up bigcouch and sv down bigcouch. Configuration, libraries, and binaries are installed into /opt/bigcouch.


Now, there is a little configuration (described in more detail in Installing & Using BigCouch).

Edit /opt/bigcouch/etc/vm.args and change:
  • -name
  • -setcookie some_secret_string
etc/defaults.ini and etc/local.ini work much like in CouchDB -- here you can fiddle with ports, enable SSL, etc. Note that the [chttpd] section describes the user-facing CouchDB server, and [httpd] describes the BigCouch "backdoor" used for administration and cluster coordination.

Your nodes will need to talk. Configure your firewall so that they can see each others' ports 5984 (CouchDB), 5986 (BigCouch), and 4369 (Erlang port mapper), plus any used for SSL.

Building the Cluster

To add a node to the cluster, you use the admin JSON API exposed on port 5986:

curl -X PUT -d {}

(And vice versa)

Now, you should be able to interact with the CouchDB server on port 5984 of either node (ignoring completely the BigCouch-ness going on) and changes should appear immediately on both nodes.

The internal sharding and replication mechanism is based on Amazon's Dynamo Paper, which is pretty generally regarded as the definitive guide to robust cluster storage.

The API to BigCouch necessarily differs a bit from CouchDB's: for example, _stats lives on the admin side since it is now cluster-wide. The differences are outlined in the BigCouch API docs.

High Availability, etc.

The nodes talk in the background to ensure consistency and all are peers, so you can connect to any one you want. For high availability, put a load balancer or other such routing in front of the nodes to create a single point of entry. BigCouch recommends HAProxy. (And for an illuminating discussion of why you might use keepalived but not heartbeat, see this post).