Clearing some hurdles automating CouchDB administration

I ran into a couple of administration issues with CouchDB while working on support for it in the excellent Pallet project ¹ , so I thought I'd leave some breadcrumbs for those that follow.

(Note that these issues were experienced with CouchDB 0.10.0 on Ubuntu Karmic. They may be resolved in later versions of CouchDB or Ubuntu, but those are the versions we're targeting for now.)

Broken Directory Permissions

First, Karmic's couchdb package is broken, insofar as key directories that CouchDB uses don't have the right ownership or mode. The symptom of this is that CouchDB will not stop properly when one invokes /etc/init.d/couchdb stop. This is a known issue, and will hopefully be resolved for Ubuntu Lucid. Rumor has it that some versions of CentOS have the same issue.

The fix is simple:

chown -R couchdb:couchdb /var/log/couchdb /var/run/couchdb /var/lib/couchdb /etc/couchdb
chmod 0770 /var/log/couchdb /var/run/couchdb /var/lib/couchdb /etc/couchdb

That's a bit of a carpet-bombing, but certainly won't do any harm, and does the trick (adjust for the install dir you have, e.g. perhaps prefixing everything with /usr/local).

CouchDB only detaches when started from a full shell

This is where the world will learn that I'm mostly an idiot when it comes to shell stuff and sysadmin in general. Thanks go to Hugo Duncan for giving me a key hint that allowed to get past this one.

In short, pallet was doing the equivalent of this in order to invoke the scripts it generates for configuration management, etc. (assuming here that your user has NOPASSWD in /etc/sudoers:

ssh -t 192.168.0.105 'sudo /etc/init.d/couchdb start'

So, we're allocating a tty, which many services need around in order to fork and detach properly (such as Tomcat via jsvc, for example). However, the CouchDB server that is started with this command dies along with the ssh session. Go ahead, give it a shot. If you really want proof, you can do this to see that the server is running before the session is closed out:

ssh -t 192.168.0.105 'sudo /etc/init.d/couchdb start;sleep 1; curl http://localhost:5984'

Of course, if you log into an environment with a full interactive session, starting CouchDB and then logging out will leave the server running as one would expect.

The solution is painfully simple in this case – just don't invoke /etc/init.d/couchdb start as an ssh exec command. Whatever you're using for configuration management, have it run in a full interactive shell session. That's exactly what Pallet is now doing for all of its configuration executions.

Results

The CouchDB crate in pallet is now pretty well bullet-proofed...or so I hope.