Migrating a native Erlang interface to RESTful Mochiweb (with a bit of TDD)

21st Century Code WorksBest of Erlang - noreply@blogger.com (Benjamin Nortier) - June 17, 2008

The title is a bit of a mouthful, but it does contain in essence what I will show you:

1. I will convert an existing native erlang CRUD interface (a simple client-server) to use a HTTP layer.
2. The HTTP calls will be RESTful.
3. I will be using Mochiweb.
4. I will show you how to do a bit of Test-Driven-Development (TDD) for this exercise using EUnit.

As an introduction, I suggest you read Steve Vinoski‘s “RESTful Services with Erlang and Yaws”, and “A RESTful web service demo in yaws” by Nick Gerakines. They are both excellent posts, and Nick’s post has a lot of interesting details, including some OTP goodness. Also check out Daryn‘s posts for a Rails twist on this topic.

I will NOT be showing much error handling, simply returning a 501 response if the REST Url has
some mistake in it.

This is the plan:
1. We will start with a working set of tests for a native CRUD interface. I will not show you the actual implementation, but it is simple enough to do yourself using a process dictionary.
2. I will show a very simple REST interface with Mochiweb, that just returns the request method type as a plaintext string. This will be tested using Inets, the native Erlang OTP internet module.
3. We will write a translators to and from plaintext and native erlang terms (again, unit tested).
4. We will combine all these into the REST interface, tested using Inets again.

Right, here is the unit tests for the CRUD interface:

crud_test_() ->   
  {setup,
   fun() -> start() end,
   fun(_) -> stop() end,
   fun(_) ->
           [
            ?_assert(ok == create(#person{id = 1})),
            ?_assert(already_exists == create(#person{id=1})),
            ?_assert(#person{id=1} == retrieve(1)),
            ?_assert(ok == update(#person{id = 1, name="Ben"})),
            ?_assert(#person{id=1, name="Ben"} == retrieve(1)),
            ?_assert(ok == delete(1)),
            ?_assert(undefined == delete(2)),
            ?_assert(undefined == update(#person{id = 2}))
           ]
   end}.

The CRUD interface is container in a module “crud”, and start() and stop() are methods to start and stop the CRUD server. For EUnit, you can define a test with a setup, teardown and tests, and this is the form that I’ve used here. There are many forms of tests for EUnit, and I suggest you consult the EUnit documentation (contained in the EUnit distribution).

EUnit automatically creates a test() function on the module when you include “eunit.hrl”, so let’s run the crud tests:

75> crud:test().
All 8 tests successful.
ok

We know that are CRUD interface is working OK.

Onto Step 2! If you’re not familiar with Mochiweb, it’s a lightweight web server developed by the guys at Mochi Media. It’s very easy to integrate into you application, and has very little configuration.

Here are the tests for the simple Mochiweb demo:

-define(URL, "http://127.0.0.1:8888").

rest_server_test_() ->
    {setup,
     fun() -> inets:start(), start_simple() end,
     fun(_) -> inets:stop(), stop_simple() end,
     fun(_) ->
             [
              ?_assert(http_result('GET') =:= "GET"),
              ?_assert(http_result('PUT') =:= "PUT"),
              ?_assert(http_result('POST') =:= "POST"),
              ?_assert(http_result('DELETE') =:= "DELETE")
             ]
     end}.

parse_result(Result) ->
    {ok, {{_Version, 200, _ReasonPhrase}, _Headers, ResultBody}} = Result,
    ResultBody.

rest_server_test_() ->
    {setup,
     fun() -> inets:start(), start_simple() end,
     fun(_) -> inets:stop(), stop_simple() end,
     fun(_) ->
             [
              ?_assert(http_result('GET') =:= "GET"),
              ?_assert(http_result('PUT') =:= "PUT"),
              ?_assert(http_result('POST') =:= "POST"),
              ?_assert(http_result('DELETE') =:= "DELETE")
             ]
     end}.

start_simple() starts the simple version of the server and stop_simple() stops it. We’re just using the base URL, and the server just returns a plain-text string of the request method. The http:request() functions are part of Inets (documentation here). You will notice that the put and post functions have extra parameters (which we will use later for the request body).

Here’s the solution:

start_simple() ->
    mochiweb_http:start(
      [{ip, "127.0.0.1"},
       {loop, {?MODULE, simple_response}}]).
stop_simple() ->
    mochiweb_http:stop().
    

simple_response(Req, Method) ->
    Req:ok({"text/plain", atom_to_list(Method)}).
simple_response(Req) ->
    simple_response(Req, Req:get(method)).

We’ve started Mochiweb, and told it that the “simple_response” function in the current module should be used to handle requests. It takes one parameter, which contains the request data. The data can be queried with different methods to extract data from it, e.g. Req:get(method), gives you the method used.

The Req:ok() function is used to respond to a request, and we simply return plain text (with the text/plain MIME type).

And let’s run the tests:

77> rest_server:test().
...
All 4 tests successful.
ok

Nice. The next bit of code we need is to translate Erlang terms to and from plain text. We’ll use Base64 encoding for this. Erlang also has very hand term to binary and binary to term functions that we can use to achieve the translation.

The tests for the translation:

term_to_plaintext_test_() ->
    A = anatom,
    B = {a, 23, "abc"},
    C = {props, [{a,3},{b,4}]},
    [
     ?_assert(A == ptt(ttp(A))),
     ?_assert(B == ptt(ttp(B))),
     ?_assert(C == ptt(ttp(C)))
    ].

Where “ttp” would be “term_to_plaintext” and “ptt” is
“plaintext_to_term”. Here’s the implementation:

ttp(Term) ->
    base64:encode_to_string(term_to_binary(Term)).

ptt(PlainText) ->
    binary_to_term(base64:decode(PlainText)).

And the test result:

77> rest_server:test().
...
All 7 tests successful.
ok

(7 tests, since we have the first 4 and now the extra 3). Now things are getting a bit more complicated. We now have to ensure that the response to the request is converted from the native Erlang terms to plaintext, this result is then returned, and we also need a function to convert the request result into the native form.

Here we go:

result_to_terms(Result) ->
  {ok, {{_Version, 200, _ReasonPhrase}, _Headers, ResultBody}} = Result,
  ptt(ResultBody).

rest_api('GET', Path) ->
    Result = http:request(get, {?URL ++ Path, []}, [], []),
    result_to_terms(Result);
rest_api('DELETE', Path) ->
    Result = http:request(delete, {?URL ++ Path, []}, [], []),
    result_to_terms(Result).

rest_api('POST', Path, Data) ->
    Result = http:request(post, {?URL ++ Path, [], [], ttp(Data)}, [], []),
    result_to_terms(Result);
rest_api('PUT', Path, Data) ->
    Result = http:request(put, {?URL ++ Path, [], [], ttp(Data)}, [], []),
    result_to_terms(Result).

crud_server_test_() ->
    {setup,
     fun() -> crud:start(), inets:start(), start() end,
     fun(_) -> stop(), inets:stop(), crud:stop() end,
     fun(_) ->
      [
       ?_assert(ok == rest_api('POST', "/person", #person{id = 1})),
       ?_assert(already_exists == rest_api('POST', "/person", #person{id = 1})),
       ?_assert(#person{id=1} == rest_api('GET', "/person/1")),
       ?_assert(ok == rest_api('PUT', "/person/1", #person{name="Ben"})),
       ?_assert(#person{id=1, name="Ben"} == rest_api('GET', "/person/1")),
       ?_assert(ok == rest_api('DELETE', "/person/1")),
       ?_assert(undefined == rest_api('DELETE', "/person/1")),
       ?_assert(undefined == rest_api('PUT', "/person/1", #person{id = 2}))
      ]
     end}.

There is a bit of duplication here, but for illustration I’ve kept the functions seperate. You will notice that the data is converted to plaintext for the PUT and POST requests. result_to_terms() converts the HTTP result from the plain text to the native Erlang terms.

You can scroll back to the original CRUD tests, and notice that the tests do exactly the same thing as on the CRUD interface, but we would have to do a “GET” + “/person/1” to get person 1, instead of doing a crud:retrieve(1).

The mapping between the CRUD and REST method are as follows:
GET <=> RETRIEVE
POST <=> CREATE
PUT <=> UPDATE
DELETE <=>DELETE

Here’s the final product:

start() ->
    mochiweb_http:start(
      [{ip, "127.0.0.1"},
       {loop, {?MODULE, crud_response}}]).
stop() ->
    mochiweb_http:stop().


crud_response(Req, 'GET', "/person/" ++ IdString) ->
    Id = list_to_integer(IdString),
    Response = crud:retrieve(Id),
    Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'DELETE', "/person/" ++ IdString) ->
    Id = list_to_integer(IdString),
    Response = crud:delete(Id),
    Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'POST', "/person") ->
    Body = Req:recv_body(),
    Person = ptt(Body),
    Response = crud:create(Person),
    Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'PUT', "/person/" ++ IdString) ->
    Id = list_to_integer(IdString),
    Body = Req:recv_body(),
    PersonWithNewValues = ptt(Body),
    UpdatedPerson = #person{id = Id, 
                            name = PersonWithNewValues#person.name, 
                            email_address = PersonWithNewValues#person.email_address},
    Response = crud:update(UpdatedPerson),
    Req:ok({"text/plain", ttp(Response)});
        
crud_response(Req, _Method, Path) ->
    Req:respond({501, [], Path}).

crud_response(Req) ->
    crud_response(Req, Req:get(method), Req:get(path)).

Once again, there is some duplication, but it’s easier to grasp when we split the functions. The GET and DELETE requests are simple to handle, since we just convert the Id string to an integer (which could throw an exception, you would have to handle that in some way), and call the CRUD interface.

POST is not much more complicated, we just get the body of the request using Req:recv_body().

The PUT function has a problem. There is duplication of the Id of the person, in both the URL “/person/1”, and the actual term, #person{id=1,...}. I’m not sure how to handle this, comments are welcome. Perhaps you can generate an error response if the Ids don’t match. Or you can make sure the posted record does NOT contain an Id field, and return an error if it does.

The solution as I’ve given it, uses the fields in the record, and ignores the Id of the record, using the Id in the URL instead.

The last function generates an error if the request could not be matched.

And the moment of truth:

81> rest_server:test().
...
All 15 tests successful.
ok

Looks so simple now, but I can asure you that it took some effort to get all the tests to pass!

Well I hope I’ve shown you something that you didn’t know before, or even encouraged you to learn some Erlang.

Comments and criticisms are most welcome :)

Categories: Blogs  21st Century Code Works  Best of Erlang  

Comments

Add comment

Name:

Email:

URL:

Smileys

Remember my personal information

Notify me of follow-up comments?