Greetings from the Bottom of the Benchmarks

I can guarantee that if you write a benchmark pitting Erlang’s dictionary type against that of any other language, Erlang is going to lose. Horribly. It doesn’t matter if you use choose the dict module or gb_trees; Erlang will still have an embarrassing time of it, and there will be much snickering and posting of stories on the various programming news aggregation sites.




Is the poor showing because dictionaries in Erlang are purely functional, so the benchmark causes much copying of data? Or perhaps because Erlang is dynamically typed?




Neither. It’s because the standard Erlang dictionary modules are written in Erlang.




In that light, the low benchmark numbers are astoundingly impressive. The code for every dictionary insert, look-up, and deletion is run through the same interpreted virtual machine as any other code. And the functions being interpreted aren’t simply managing a big, mutable hash table, but a high-concept, purely functional tree structure. The Erlang code is right there to look at and examine. It isn’t entangled with the dark magic of the runtime system.




There are still some targets of criticism here. Why are there multiple key/value mappings in the standard library, one with the awkward name of gb_trees, in addition to the hackier, clunkier-to-use “Erlang term storage” tables? Why would I choose one over the others? Why is dict singular and gb_trees plural? Let’s face it: The Erlang standard library is not a monument of consistency.




But performance? I’ve used both dictionary types in programs I’ve written, and everything is so instantaneous that I’ve never taken the time to see if a disproportionate amount of time is being spent in those modules. Even if I’m overstating things, over-generalizing based on the particular cases where I’ve used dictionaries, it’s still high-level Erlang code going up against the written-in-C runtime libraries of most languages. And that it comes across as “instantaneous” in any kind of real-world situation is impressive indeed.




(If you liked this, you might enjoy Tales of a Former Disassembly Addict.)

Comments

Consistency and Performance…

You are quite right…

The Erlang standard library could really do with a makeover on the consistency department. It wouldn’t take too much effort to re-market stdlib with new modules that provide a thin layer over the existing modules. Those modules would still exist, but be “hidden”. I think it’s one of the reasons for the slower than I would like take up of Erlang. Strings is another good one: Should I use lists or binaries? Where is one better than the other? And OTP…can we change the name…please???

As for performance, well let’s be honest no one claims for doing work that you would otherwise do in Java or C++ one would chose Erlang. Erlang’s strength is as a control language - in that it’s very fast. However, you are quite right, it does make Erlang look bad when someone does a benchmark, chooses gb_trees or dict instead of ETS and gets awful performance. A manager sees that and runs a mile…Not sure I have an answer to that: wait for LLVM and natively compile all modules? Re-write common modules as a BIF (like the lists module was done), or do something like digraph and use ETS to save/represent data and have the module as a wrapper over ETS.

Posted by Matt Evans on 13 Oct 2011 at 15:38

I had been very happy to visit this url I seriously like to suggest the following to absolutely everyone because theres no doubt does nono hair removal work concerning it’s articles.

Posted by kaseyr on 27 Jan 2012 at 14:00

Vendor benchmarks tend to ignore requirements for development, test, and disaster recovery computing capacity. Vendors only like to report what might be narrowly required for production capacity in order to make their initial acquisition price seem as low as possible. Thanks a lot.
Regards,
Personal statement writers

Posted by jaffa on 04 Feb 2012 at 07:22

Add comment