CouchDB Benchmarks: gcc vs. clang

Hypothetical Labs - kevin - September 06, 2009

Since the release of Snow Leopard, I’ve been itching to run some benchmarks. Jan Lehnhardt suggested I give the benchmarking script from this bug a few runs. So that’s what I did.

At first glance testing seemed pretty simple. Compile one version of Erlang with gcc and another with clang. Then use each of those versions to compile CouchDB and run benchmarks against each Erlang/CouchDB pair. Since CouchDB also has at least one bit of native code, couchjs, I’d also used gcc and clang to compile that, too.

Then reality showed up and ruined my day. Compiling CouchDB on OS X is, well, a pain in the ass involving MacPorts and lots of dependencies if you do it wrong.  Doing it right means grabbing Jan’s couchdbx-core project from Github and building new guts for CouchDBX. This is the way to go, IMHO. 10 - 15 minutes’ of compiling gave me a new Erlang, SpiderMonkey, and CouchDB I could drop into the Contents/Resources/couchdbx-core directory of my previously installed CouchDBX app.

Compiling with clang is similarly easy. All you need to do is set the environment variable CC to point to /Developer/usr/bin/clang. I found I also needed to set CFLAGS to -Qunused-arguments to get Erlang to compile.

The benchmark itself consists of three phases. The first phase creates a database and populates it with 40,000 documents. The second phase creates and executes a complex view named ‘megaview’ to really exercise the view collation code. The last phase creates and executes a simple view named ’simpleview’.

Let me quickly describe how the performance numbers were collected. The benchmark was executed 5 times against the clang and gcc versions of CouchDB. I’ve computed a raw average across all runs as well as a “normalized” average which throws out the highest and lowest times. Finally, I computed the standard deviation as a indicator of overall variability.

Enough talk. Here are the numbers:

Note: Document insertion times are given in seconds while view collation numbers are given in minutes

Operation Average Normalized Average Standard Deviation
Insert 40k docs (clang) 41.27 41.28 0.048
Insert 40k docs (gcc) 41.02 41.02 0.132
megaview (clang) 37.68 37.68 0.664
megaview (gcc) 17.97 17.95 0.463
simpleview (clang) 1.59 1.4 0.442
simpleview (gcc) 0.75 0.74 0.020

The megaview test really highlights the disparity between clang and gcc. It’s worth noting the test also showcases a bottleneck in CouchDB and only used a single core on my laptop. If I had to guess I’d say the performance disparity is a function of the age of each compiler. gcc has been around forever so it makes sense it’s had more time to implement more/better optimizations than clang.



Categories: Blogs  Hypothetical Labs  

Comments

No comments so far, you could be the first.

Add comment

Name:

Email:

URL:

Smileys

Remember my personal information

Notify me of follow-up comments?