PubSub-over-Webhooks with RabbitHub. RabbitMQ, the Erlang-powered AMQP message queue, is growing an HTTP interface based on webhooks and PubSubHubBub.

When moving with your iPhone, as a pedestrian, bike rider, car driver or in public transportation, you might have to shutdown the OneTeam for iPhone application while staying connected to your XMPP server. It is convenient for battery saving purposes, or because you want to use another application (iPhone OS 3.0 still does not authorize background applications).

When an event occurs on your XMPP server, it is sent to the APNS (Apple Push Notification Service), which instantly relays it to your iPhone. A push notification is then poped up on your screen, with a sound played. The OneTeam badge (icon) is then changed, showing a new event has arrived, or the number of events waiting for you to read.

You simply have to tap the OneTeam icon to read your new event. The reconnection will be fast and seamless.

This feature is very handy, since you can use your iPhone the way you want (browse the web, use another application) while still being instantly notified as if you were using OneTeam on the background.

This push notification feature is also deployed on OneTeam.im XMPP server, and ejabberd has a component, IMpush, that can send push notifications to the APNS. This makes ejabberd and OneTeam the first XMPP client and server solution that supports the Apple push system.

OneTeam for iPhone has also received fixes and improvements: groupchat, file transfer, image sending… Here are a few screenshots:

An example of Groupchat:

See the Push configuration screen:

And the general OneTeam confguration:

A few comments on the experience after having used OneTeam for iPhone with push and groupchat:

• “Battery life is not impacted by being always connected to my XMPP server on mobile. I have used all mobile XMPP client and this is the first time I can truely stay connected all the time. I feel this is really really the premise of the mobile revolution for mobile instant messaging and mobile XMPP.“
• “OneTeam for iPhone client has the type of feature you would expect from a good desktop XMPP client and not a mobile one. You can use groupchat, file transfer. This is really what I need professionally.“

OneTeam for iPhone will be available on the Apple’s App Store in a few days, after the usual moderation period at Apple.

IMpush is sold as soon as today on the IMstore to prepare ejabberd servers ahead of OneTeam 3.0 with push.

erlang:lists(CouchDBNaked) - Harish Mallipeddi, a performance engineer at Yahoo, has put together a great post on the internals of CouchDB, and how to even use some of the couch source to build your own B-Tree based mini-application.

erlang:lists(ErlangFactory) - Slides and videos are up from the 2009 London Erlang Factory conference. About half have videos and most have slides available. Not sure yet if they will all have videos or if there are reasons for not showing the films for some speakers.

Each week I’ll mention a few must-see articles, blog entries or videos. If you have something to bring to my attention - contact me at chad at inakanetworks com.

It will be available in couple of days.

]]> Blogs, Caoyuan Blog 2009-06-30T18:19:00+01:00 http://www.planeterlang.org/en/planet/article/ProcessOne_presentations_at_Erlang_Factory/ http://www.planeterlang.org/en/planet/article/ProcessOne_presentations_at_Erlang_Factory/ Mickaël Rémond There have been two presentations from ProcessOne at Erlang Factory. Here are the slides of the presentations.

The first presentation by me (Mickaël Rémond) was titled “OneTeam Media Server: Adding Video to Instant Messaging with Erlang”.

The second presentation by Geoff Cant was title “Whitelabel Erlang” and give feedback on how Erlang can be used to write scalable hosted application for business.

Installing daemontools is really simple on debian. You just create an Installation Script which sets up the files the supervise instance later uses to control your service. Save the following script to install_ejabberd_svc.sh and make it executable.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

#!/bin/bash
# v2009-04-09 tp@turtle-entertainment.de
# ejabberd and daemontools
#
echo STARTING EJABBERD WITH DAEMONTOOLS
# see
# http://cr.yp.to/daemontools/svc.html
# http://cr.yp.to/daemontools/softlimit.html
# http://www.kdegraaf.net/supervise.html
#
SERVICEDIR=“/etc/service”
SERVICE=“ejabberd
# No svc should be running, but to make sure we let it die first
svc -d ${SERVICEDIR}/${SERVICE} &> /dev/null
test -d && rm -rf ${SERVICEDIR}/${SERVICE}
mkdir -p ${SERVICEDIR}/${SERVICE}
test -f && rm -f ${SERVICEDIR}/${SERVICE}/run
test -f && rm -f ${SERVICEDIR}/${SERVICE}/tmprun
cat > ${SERVICEDIR}/${SERVICE}/tmprun <<EOF
#!/bin/sh
exec 2>&1
sleep 3
exec /opt/ejabberd/server/sbin/ejabberdctl start
EOF
chmod 755 ${SERVICEDIR}/${SERVICE}/tmprun
mv -f ${SERVICEDIR}/${SERVICE}/tmprun ${SERVICEDIR}/${SERVICE}/run
# Create log Pipe
mkdir -p ${SERVICEDIR}/${SERVICE}/log
chmod 1755 ${SERVICEDIR}/${SERVICE}/log
test -f && rm -f ${SERVICEDIR}/${SERVICE}/log/run
test -f && rm -f ${SERVICEDIR}/${SERVICE}/log/tmprun
cat > ${SERVICEDIR}/${SERVICE}/log/tmprun <<EOF
#!/bin/sh
exec env - PATH=“/usr/bin” logger -p local5.err -t EJABBERD
EOF
chmod 755 ${SERVICEDIR}/${SERVICE}/log/tmprun
mv -f ${SERVICEDIR}/${SERVICE}/log/tmprun ${SERVICEDIR}/${SERVICE}/log/run

Now you just need to run this script and install the daemontools-run package under debian. This automatically uses the just created structure and starts ejabberd with logging stdout to logger. Keep in mind that every programm you run under daemontools must run in foreground.

1

aptitude install daemontools-run

Thats it!

If you want to use rsyslog just setup your rsyslog.conf for redirecting local5.err to your rsyslogserver

1

local5.err @10.10.0.99:514

In this Analyst Opinion series by Daniel Okubo, it is observed that social networks and mobile network operators are currently deploying IM and presence features based on XMPP.

In a competitive and recessive context, the leaders and newcomers look to differentiate in order to increase their market shares. Real time communication based on open standard like XMPP has proven a good way to value existing users base.

This document is available at Datamonitor ResearchStore, at the price of $150, or $375 for a multi-user license. ProcessOne is not affiliated with DataMonitor and does not earn any commission on sales of this document, but we thought it could be usefull to know for customers interested in ProcessOne.

I’ll be visiting London, Amsterdam, Zurich and the Gran Canaria. Here’s when, how and why:

June 22nd–26th: CouchDB University & Factory, London

The CouchDB University is a a three day training course where J Chris and I teach a select group of students everything about CouchDB. With little prior knowledge, we’ll leave you with being able to build amazing CouchDB applications at small and large scale as well as extend CouchDB itself.

The CouchDB Factory is a track at the Erlang Factory running all day Friday.

June 29th–30th: Kings of Code, Amsterdam

Kings of Code looks like it is going to be a kick-ass web developer conference featuring some of my favourite web people: Geoffrey Grosenbach Joe Stump & Francisco Tolmasky. I’m fairly confident that the other speakers will be among my favourites after Kings of Code :)

J Chris will be talking about CouchDB.

It’s still in discussion, but I might talk about CouchDB and Erlang for web developers on one of the side events.

July 1st–2nd: ICOODB, Zurich

I’ll be taking the night train from Amsterdam zu Zurich to give a three hour tutorial as well as a 60 minute presentation on CouchDB at the International Conference on Object Databases. CouchDB is strictly not an object oriented database, but it stores objects and is of interest to the research community that meets in Zurich.

Prof. Stefan Edlich invited me to speak at ICOODB and I’m very happy I can make it.

July 1st–7th: GUADEC, Gran Canaria

Canonical, the kind folks behind the Ubuntu Linux distribution are pushing CouchDB to become a centerpiece of the Ubuntu desktop data synchronization infrastructure. Merrily sync your contacts, calendar data between your machines, an online backup service and share select data with your peers. And yeah UbuntuOne is also related :)

Canonical is flying me out to attend the Linux Desktop Summit to talk to desktop application developers and show them how cool CouchDB is and where it is useful for them.

Also, Gran Canria, I couldn’t say no. Thank you Canonical!

As much as I am excited about the travels and meeting all you out there, I’ll be missing three weeks in my favourite city, Berlin and it makes me a little sad.

Well, I didn’t mind it until I had to build one of my drivers on Windows. It’s been a long time since I’ve done much Win32 development but I used to know my way around Visual C++ pretty well. “I used to do this shit for a living. How hard could this be?”, I thought to myself.

Three words: famous last words.

Three more words: C++, linker, XML.

Two final words: sheer frustration.

After wrestling with Visual Studio Express C++ Edition 9.0 (or whatever crazy name MS uses) for a day and a half I finally have a working driver. Here’s the nuggets o’ knowledge I learned along the way. These notes assume you’re porting a working driver from OS X/Linux/BSD to Windows although most of them are applicable if you’re starting from scratch on Windows, too. They also assume you’re building code from inside of Visual Studio because, frankly, I don’t care enough about Windows to figure out nmake, too.

• Start with a blank Win32 DLL project. Don’t let Visual Studio create stdafx and friends. Down that path lies madness. You’re much better off to create a blank project and import your code.
• Add the erts-<version_number>/include directory to your project’s include path. You’ll need this so the compiler can resolve erl_driver.h.
• Don’t forget to include string.h before erl_driver.h. erl_driver.h uses memcpy() without including string.h. Including string.h first will fix a compiler warning and a possible runtime crash.
• Very Important: Right-click your project and navigate to the Linker -> Manifest File option.  Enter this magical text into the Additional Manifest Dependencies field:

  type=‘win32’ name=‘Microsoft.VC90.CRT’ version=‘9.0.21022.8’ processArchitecture=‘x86’ publicKeyToken=‘1fc8b3b9a1e183b’
  

  I believe this tells the compiler to link the DLL with the Visual C++ 9.0 runtime. This is the equivalent of the C stdlib on other platforms, I think. Erlang will be unable to load your driver if you skip this step.

---

Observation: Who in their right fucking mind thought it was a good idea to add XML descriptors to the link step of building a C library?! I take back every bad thing I’ve ever said about Java’s use of XML. SOAP, with all of its angle brackety warts, almost looks sane in comparison.

---

• With the previous step accomplished you’ll need to distribute the VC++ runtime with your driver. Luckily, MS provides the code in a redistributable form here inside the Microsoft Visual Studio 9.0VCredistx86 directory structure. Grab all of the files in there and plop them into the same directory as your driver’s DLL file.

There are probably better ways to accomplish the same goal given my dated platform knowledge. If you know of a better way to do this, please drop a comment and set me straight!

One of the drivers I’ve been working on is to allow SpiderMonkey to interface with Erlang. I’ve asked and am waiting for permission from my client to open source the code since I think it’d be generally useful. If that doesn’t pan out I’m strongly considering doing another driver — open source from the beginning — to interface with V8 or possibly WebKit’s engine. I’ve got some ideas on how JSON could smooth the interop between the two languages. Anyone up for either working on or sponsoring work on this?

NetBeans 6.7 RC3 or above is a requirement.

What’s new:

• It’s rewritten in Scala instead of Java
• More reliable instant rename
• Display extracted document information from source comment when doing auto-completion.

To download, please go to: https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=226387

To install:

• Open NetBeans, go to “Tools” -> “Plugins”, click on “Downloaded” tab title, click on “Add Plugins…“ button, choose the directory where the Erlang plugin are unzipped, select all listed *.nbm files, following the instructions.
• Make sure your Erlang bin path is under OS environment PATH, you can also check/set your OTP path: From [Tools]->[Options], click on ‘Miscellanous’, then expand ‘Erlang Installation’, fill in the full path of your ‘erl.exe’ or ‘erl’ file. For instance: “C:/erl/bin/erl.exe”
• When you open/create an Erlang project first time, the OTP libs will be indexed. Take a coffee and wait, the indexing time varies from 10 to 30 minutes depending on your computer.

Feedback and bug reports are welcome.

]]> Blogs, Caoyuan Blog 2009-06-17T17:08:13+01:00 http://www.planeterlang.org/en/planet/article/Webmachine_Bloggy_Goodness/ http://www.planeterlang.org/en/planet/article/Webmachine_Bloggy_Goodness/ Webmachine (and Erlang use in general) is picking up. If you’re looking for a few more people to follow in relation to webmachine, I have some suggestions.

If you haven’t, check out part one.

Say you want to find out what’s behind the buzz of all these new #nosql databases. There’s a large number to choose from today. All options come in varying degrees of maturity and characteristics so it’d be nice to know what solves your problem best. A non inclusive list of these databases or storage systems include Memcache[DB], Tokyo Cabinet / Tyrant, Project Voldemort, Scalaris, Dynamite, Redis, Persevere, MongoDB, Solr or my favourite CouchDB. And these are just some of the open source ones.

This article is not a comprehensive comparison of any of the mentioned systems. Instead it tries to give you an idea about what to look for when evaluating a storage system or how to take into perspective evaluations and benchmarks others have done.

We’ll look at some of the technical aspects of data storage systems: Applying common sense when reading benchmarks; b-trees and hashing; speed vs. concurrency; networked systems and their problems; low level data storage (disks’n stuff); and data reliability on single-nodes and multi-node systems.

There are a lot of other reasons to decide for or against a project based on a lot of non-technical criteria, but things like commercial support or a healthy open source community are not part of this article.

Astounding Numbers

From time to time you see some crazy numbers posted to the reddits of the internets that claim fantastic performance.

“The (imaginary) SuperfastDB can store 450,000 items per second!”.

Wow.

No word on where the items are stored (in memory? on a harddrive? Spindles? Solid State?), what an item is exactly and how big it is, the rest of the hardware this was run on and how to reproduce it.

But boy, 450,000 a second!

My shoes can do 650,000 a second, but you’ve got to figure out what.

Context is as important as reproducibility. The last article here established that finding out that my system and your system come up with different numbers is not much of a help. Any sort of serious test must come with a set of scripts or programs and comprehensive instructions on how the tests were run.

Everything “cool” in computer science has been around for 25+ years. Actual innovation is rare. Advancements in hardware and new combinations of existing solutions make for new stuff coming out each day (that’s a good thing), but the fundamental rules are the same for all. We’re all running von Neumann machines, quicksort is still pretty quick and hashes and b-trees rule the storage world.

Let’s recap.

Hashes & Trees

Hashing revolves around the idea of O(1) lookups. Allocate a number of buckets, create a function that gives you a number of a bucket for any data item you might want to store, make sure no two data items hit the same bucket (or work around that). Runtime characteristics include that you only need to ask your function where to look for or store your data and the allocation of your set of buckets: If you need to store more items than you have buckets, some more work is required which gives you O(N)` operations that you can’t ignore in practice.

The other elephant in the room are b-trees. The fundamental idea here is to get to your data in a minimal number of steps traversing a tree because making a step is expensive, but reading your data is very fast comparatively. Steps are expensive because they translate to a head seek (that is the time your spinning hard drive needs to position the reading arm to find the spot to read your data from), but reading from a harddrive once the reading head is in place is fast.

There are a bunch of more interesting lookup structure like R-Trees for spacial queries, but they are mostly used for secondary indexes on top a regular data set that lives in a hash or b-tree.

Concurrency vs. Speed

Concurrency is hard. The devil lies in the details and when briefly looking at things, the details are often overlooked. Suites the devil.

Creating storage systems that assume only one access occurs at a time is relatively easy. If resources are shared concurrently, things become tricky. The two larger schools of thought (and practice) are locking and no-locking (heh).

Locking means that the database has to maintain information for everybody who wants to write to a part of the database, and what part it is.

No locking, or optimistic locking or MVCC moves that burden to the person who is trying to write to the database. She must prove that she won’t be overwriting any existing data.

The trade-offs here are a leaner request handing on the server that works well with remote & concurrent clients at the expense of more complexity on the client (the person who wants to store something in our database).

Hybrid approaches are possible too: While MVCC is used internally, the database’s clients can rely on database-side locking (e.g. PostgreSQL or InnoDB).

Networks

Just a quick note: We already talk about client and server here. There is a strong case for embedded databases like SQLite that don’t expose a concurrent user model to the outside. The program that needs an embedded database just includes it.

Another approach to using databases is having a dedicated computer running a database system and sharing it over the network with any number of clients using this database server. They can often be “a bunch of servers” or a cluster. More on that later.

A separate database server (networked or not) will need to spend some time to deal with connections, network failures, unspecified client behaviour and so on. The upside is a piece of infrastructure that can be maintained separately. An embedded database will thus be faster but probably won’t solve all of your problems and it will always be tied to your application.

fsync(): Reliability vs. Speed

When people tell me “SuperfastDB does 450,000 a second!” I ask “How many fsync()s is that?”. Let me explain:

A database system uses operating system services to use any hardware. The operating systems exposes a harddrive through a filesystem. The database systems talks to the filesystem and asks it to store or retrieve data in its behalf. The filesystem then goes ahead and tries to satisfy the database’s requests.

(I’ll not talk about databases that can use raw block devices to store data. They exist but they are not as common as those who use the filsystem.)

The filesystem also tries to be clever – for good reasons. When the database requests a piece of data, the filesystem will not only find that piece and return it, it will also store it in a cache to avoid having to actually talk to the harddrive the next time this piece of data gets requested. When the data changes, the filesystem either removes it from the cache or updates it with the harddrive. It might even go further and only store the new data that comes in with a write request into the cache and rely on a periodic task to write all of the cache back to the drive. Writing a bunch of of pieces at once is more efficient than storing each one on its own.

More efficient equals to faster and faster is good, right? Well, it depends: If all goes well, this approach is a nice one. But you know computers, things will not go well 100% of the time. The failure scenarios are endless, but they boil down to the question: “What happens when your machine dies and you have data that has only been written to memory?” — The answer isn’t too hard: That data is lost. If there is a delay between a write request finishing and data being written (or “flushed”) to disk any data that has been “written” during the delay period is subject to lost.

There are cases where this is not a problem; in other cases it is. A developer should have the chance to decide. (Note that even your hardware could be lying to you about having stored data, but I’ll punt on this one, get proper hardware).

So, flushing to disk needs to happen before you can rest assured your data has been stored. Your operating system has an API call that forces the filesystem to write its cache to disk. It is called fsync() (on UNIX systems) and it is an expensive operation. You can only do so many fsync()s in a second and it is not a great many.

The 450,000 items were most likely just written to memory and not to disk.

Space & In-Place

When writing files to disk (at the end of the day, your data ends up in one file or another on the filesystem) that represents what lives in a database, there are multiple options to handle updates.

An update is a change to your data item, for example, a new phone number. The intuitive way to handle this is to go and find the old phone number in the file, and overwrite it with the new number. Easy.

There are several problems with this approach: What to do if the new phone number is longer than the old one (say you added an international calling prefix)? The new number needs to be written to a different place and the change in location must be recorded. Not too big of an issue.

Back to failure scenarios: Again, the reasons can be manifold, but what happens when we’ve (over-)written the first 4 digits of the old with the new number and then the server dies, power goes away or the database server crashes? The next time you want to read the phone number you get a mix of the old and the new one (if you are lucky) and you don’t exactly know that this is the case and which parts are missing. Your database file is inconsistent and you need to run a integrity check to find missing bits and correct half-written bytes. In the worst case that means scanning your entire database file a few times before you resolved all inconstancies. If you have a lot of data, that can take days.

To solve this, you always write the new phone number to a new place in the database file and only when it has been fsync()ed to disk, you update the location of the phone number (and then flush that update to disk as well). You will never end up in a scenario where your database file can end up an inconsistent state and after a crash you are back online without an integrity check.

The trade-off for consistency is write-speed (remember fsync()s are expensive) for consistency-check-speed after a failure.

A nice bonus is that if the “new place in the database” is the end of the file, you keep your disk-drive head busy with writing data to disk instead of seeking all over the place (remember: seeks are expensive).

Distribution, Sharding & Resharding

So far, we’ve been looking at scenarios that involve a single database. We learned a great deal (I hope), but in reality we often deal with more than one database. The simplest reason to have two databases is for redundancy. Failures can bring down your database temporarily or even permanently. If it is a temporary issue, waiting a bit (or a bit longer) to get up and running again might be an option, but often, an application or service should be available at all times. A fatal failure where a database server is lost beyond repair, your data is gone if you haven’t stored it in a second place.

“I’ll just make two copies, easy!”. Yup easy, until you look at the details (that damn devil again!).

It’s all about failures again. Consider a single read request. A client connects to a server and asks for a data item. The server looks it up and returns the data to the client. All is well. At any point things can go wrong. The network connection can drop (or slow down so much that client or server assume it dropped), the client can disappear (because of a network failure or crash) as can the server. Clients, servers and the protocols they speak need to be built around the assumption that any of these things (and many more) can go wrong. If any parts is not designed to handle error cases, your system will do funny things, but it won’t reliably store and manage your data.

Add complexity: With each write target (store in two places) the possibility of error and the need for proper error handling grows exponentially. When evaluating a distributed storage system, looking at how errors are handled is vital.

Another reason to distribute data among multiple servers is capacity. The three metrics of interest here are read requests, write requests and data. If you have more requests or data than a single machine can handle, you need to move to multiple machines. Each metric calls for different strategies, but they often go along with each other. The need for fault tolerance that I discussed above needs to be considered alongside.

Growing read capacity is relatively easy once you covered the base case where the source for reading data might not be the same as the the target for writing data and that there can be a mismatch (cf. eventual consistency).

Distributing writes and data works by designating two machines with 50% of the operations. A clever intermediate, a proxy server for example, decides which request goes where and all is well, we can store twice as much and we can store at twice the speed. When we need to grow bigger yet, we add another server and tell the proxy server to distribute the load equally among them. Adding a proxy for distribution introduces a single point of failure and you don’t want these; there’s added complexity with this approach.

The diagram shows that there is another step needed that wasn’t included in the above description. The new “node” needs to have a copy of all data items that are assigned to him and are currently living on the two existing nodes. The process of moving data items to new nodes is called resharding and needs to happen every time a new node is added.

Resharding can be an expensive operation if you have a lot of data. Techniques like consistent hashing help with minimising the amount of items that need to move. If you are looking at a sharding database, you want to understand how the sharding is performed and if you like the trade-offs.

CAP Theorem

The CAP Theorem states that out of consistency, availability and partition tolerance, a system can choose to support two at any given moment, but never three.

Consistency guarantees that all clients that talk to cluster of nodes will always get to read the same data. Write operations are atomic on all nodes.

Availability guarantees that in any (reasonable) failure scenario, clients are still able to access their data.

Partition tolerance guarantees that when nodes in the cluster lose their network connection and two or more completely separated sub-clusters emerge, the system will still be able to store and retrieve data.

Please Talk! (To Developers)

If you are aiming for a comparative benchmark of two or more systems, you should run your procedure by they authors. I found developers are happy to help out with benchmarks by clearing up misconceptions or sharing tricks to speed things up (which you can choose to ignore, if you are looking for out-of-the box comparison, but this is rarely useful).

information and trading software for the betting industry.

We are looking to hire an experienced Erlang developer to work on extending and

improving an existing trading system.  This role will involve working as part

of a small team of Erlang developers and working closely with Python developers

and the trading team.

You should have experience of developing reliable, fault tolerant

applications in

Erlang.  Experience with UNIX, networking, PostgreSQL and Python would also be

highly desirable.

* Starting salary £30,000 plus, depending on experience.

* 25 days holiday per year.

* Discretionary bonus scheme.

* Flexible working hours.

reverse(L) -> reverse(L, []).
reverse([H|T], Acc) ->
   reverse(T, [H|Acc]);
reverse([], Acc) ->
   Acc.

Now suppose there’s another function that uses reverse at the very end, just before returning:

collect_digits(L) -> collect_digits(L, []).
collect_digits([H|T], Acc) when H >= $0, H =< $9 ->
   collect_digits(T, [H|Acc]);
collect_digits(_, Acc) ->
   reverse(Acc).

This function returns a list of ASCII digits that prefix a list, so collect_digits(“1234.0”) returns “1234”.  And now one more “suppose”: suppose that one time we decide that we really need to process the result of collect_digits backward, so we do this:

reverse(collect_digits(List))

The question is, can the compiler detect that there’s a double reverse?  In theory, the last reverse could be dropped from collect_digits in the generated code, and each call to collect_digits could be automatically wrapped in a call to reverse.  If there ends up being two calls to reverse, then get rid of both of them, because it’s just wasted effort to double-reverse a list.




With lists:reverse as a built-in, this is easy enough.  But can it be deduced simply from the raw source code that reverse(reverse(List)) can be replaced with List?  Is that effort easier than simply special-casing the list reversal function?