Webmachine

Cowboy and Batman.js for Erlang Web Development

Why Cowboy and Batman.js

There are a lot of Erlang web frameworks out there today. Not all are modeled after the MVC model (see Nitrogen), but I think all of them are addressing the problem the wrong way. I recently gave a presentation, slides here and the code for this example here, describing my perferred method for using Erlang for web development and why I think it is the best way to go. In this post, I'll go into more details on how to build the Erlang backend for the TodoMVC clone I did with Batman.js. I will not spend time on Batman.js but instead only give a quick list of reasons I prefer it to other Javascript frameworks.

Batman.js advantages:

  • Automatic URL generation based on model
  • HTML data-bind templates
  • Coffeescript
    Cowboy is a newer Erlang web server that provides a REST handler based on Webmachine. Both of these are perfect for developing a RESTful API, because they follow the HTTP standard exactly and when you are building an API based on HTTP, being able to properly reason about how the logic of the application maps to the protocol eases development and eases getting REST "right".

Nginx

Any non-dynamic content should be served by Nginx since there is no logic needed and it is something Nginx is great at, so why have Erlang do it? The snippet below configures Nginx to listen on port 80 and serve files from bcmvc's priv directory. Each request is checked to see if it is a POST or any other method with a JSON request type. If either of those are true, the request is proxied on to a server listening on port 8080, in our case the Cowboy server.

server {  
  listen 80;  
  server_name localhost;  

  location / {  
    root   <PATH TO CLONE>/bcmvc/lib/bcmvc_web/priv/;  

    if ($request_method ~* POST) {  
      proxy_pass        http://localhost:8080;     }  

    if ($http_accept ~* application/json) {  
      proxy_pass        http://localhost:8080;     }  
  }  
}

The API

Batman.js knows what endpoints to use and what data to send based on the name of the model we created and the encoded variables, code here. This results in the following API:

 
Method Endpoint Data Return
POST _todos_ {todo : {body:"bane wants to meet, not worried",isDone:false}}
PUT /todos/33e93b30-2371-4071-afc5-2d48226d5dba {todo : {body:"bane wants to meet, not worried",isDone:false}}
GET _todos_ [{todo : {id:"33e93b30-2371-4071-afc5-2d48226d5dba", body:"bane wants to meet, not worried", isDone:false}}]
DELETE /todos/33e93b30-2371-4071-afc5-2d48226d5dba

Cowboy Dispatch and Supervisor

Dispatch rules are matched by Cowboy to know what handler to send the request to. Here we have two rules. One that matches just the URL /todos and one that matches the URL with an additional element which will be associated with the atom todo. Both requests will be sent to the module bcmvctodohandler.

Dispatch = [{'_', [{[<<"todos">>], bcmvc_todo_handler, []},  
                   {[<<"todos">>, todo], bcmvc_todo_handler, []}]}],

Cowboy provides a useful function childspec_ for creating a child specfication to use in our supervisor. The child spec here tells Cowboy we want a TCP listener on port 8080 that handles the HTTP protocol. We additionally provide our dispatch list for it to match against and pass on requests.

ChildSpec = cowboy:child_spec(bcmvc_cowboy, 100, cowboy_tcp_transport,  
                              [{port, 8080}], cowboy_http_protocol, [{dispatch, Dispatch}]),

Cowboy Handler

Now that we have a server on port 8080 that knows to send certain requests to our todo handler, we can build the module. The first required function to export is init/3. This function let's Cowboy know we have a REST protocol, this is how it knows what functions to call (some have defaults and some existing in our module) to handle the request.

init(_Transport, _Req, _Opts) ->  
    {upgrade, protocol, cowboy_http_rest}.

Knowing that this is a REST handler Cowboy will pass the request on to allowedmethods/2_ to find out if our handler is able to handle this method. Next, the content types accepted and provided by the handler are checked against the incoming request. The expected HTTP response status codes are returned if any of these fail. 405 for allowedmethods, XXX for _contenttypesaccepted and XXX for contenttypesprovided.

allowed_methods(Req, State) ->  
    {['HEAD', 'GET', 'PUT', 'POST', 'DELETE'], Req, State}.  

content_types_accepted(Req, State) ->  
    {[{{<<"application">>, <<"json">>, []}, put_json}], Req, State}.  

content_types_provided(Req, State) ->  
    {[{{<<"application">>, <<"json">>, []}, get_json}], Req, State}.

Now the request is sent to the function that handles the HTTP method type of the request.

For a POST, a request to create a new todo item, the function processpost/2_ is sent the request. Here we retrieve the body, a JSON object, from the request, convert it to a record and save the model. We'll see how this record conversion is done when we look at the model module. To inform the frontend of the id of our new resource we set the location header to be the path with the id.

process_post(Req, State) ->  
    {ok, Body, Req1} = cowboy_http_req:body(Req),  
    Todo = bcmvc_model_todo:to_record(Body),  
    bcmvc_model_todo:save(Todo),  

    NewId = bcmvc_model_todo:get(id, Todo),  
    {ok, Req2} = cowboy_http_req:set_resp_header(  
                   <<"Location">>, <<"/todos/", NewId/binary>>, Req1),  

    {true, Req2, State}.

For this handler we expect PUT for an update to an object, that is what Batman.js does, but a PATCH would make more sense. For a PUT the URL contains the id for the todo item to be updated. That is retrieved with the_ binding/2_ function. The todo record is created the same as in processpost/2_ but then the this id is set for the model and the update/1 function is used to save it to the database.

put_json(Req, State) ->  
    {ok, Body, Req1} = cowboy_http_req:body(Req),  
    {TodoId, Req2} = cowboy_http_req:binding(todo, Req1),  
    Todo = bcmvc_model_todo:to_record(Body),  
    Todo2 = bcmvc_model_todo:set([{id, TodoId}], Todo),  
    bcmvc_model_todo:update(Todo2),      
    {true, Req2, State}.

For a GET request, which for this application we do not deal with a request for a single todo item, all todo items are retrieved from the model module. Each of these is passed to the model's tojson/1_ function and the result of converting each to JSON is combined into a binary string and placed between brackets so the Batman.js frontend receives a proper JSON list of JSON objects.

get_json(Req, State) ->  
    JsonModels = lists:foldr(fun(X, <<"">>) ->  
                                 X;  
                            (X, Acc) ->  
                                 <<Acc/binary, ",", X/binary>>  
                         end, <<"">>, [bcmvc_model_todo:to_json(Model) || Model <- bcmvc_model_todo:all()]),  

    {<<"[", JsonModels/binary, "]">>, Req, State}.

And lastly, DELETE. Like in PUT the todo item's id is retrieved from the bindings created based on the dispatch rules and this is passed to the model's delete function.

delete_resource(Req, State) ->  
    {TodoId, Req1} = cowboy_http_req:binding(todo, Req),  
    bcmvc_model_todo:delete(TodoId),  
    {true, Req1, State}.

Models

Model's are repsented as records and must provide serialization functions to go between JSON and a record. Each model uses a parse transform that creates functions for creating and updating the record. The transform is a modified version of exprecs from Ulf Wiger that also uses the type definitions in the record to ensure when setting a field that it is the correct type. For example in the todo model isDone is a boolean, so when the model is created the boolean convert function will be matched to convert the string representation to an atom:

convert(boolean, <<"false">>) ->  
    false;  
convert(boolean, <<"true">>) ->  
    true;

So the key pieces of the_ bcmvcmodeltodo_ are:

-compile({parse_transform, bcmvc_model_transform}).  

-record(bcmvc_model_todo, {id = ossp_uuid:make(v1, text) :: string(),  
                           body                          :: binary(),  
                           isDone                        :: boolean()}).  

to_json(Record) ->  
    ?record_to_json(?MODULE, Record).  

to_record(JSON) ->  
    ?json_to_record(?MODULE, JSON).

The_ ?recordtojson_ and ?jsontorecord macros are defined in jsonerl.hrl. These marcos are generic and work for any record that is typed and uses the model transform.

Conclusion

Clearly, much of what the resource handler and model do is generic and can be abstracted out so that implementing new models and resources can be even simpler. This is the goal of my project Maru. Currently it is based on Webmachine but is now being convered to Cowboy.

In the end, using Cowboy for building a RESTful interface for your application allows you to build interfaces for the frontend entirely separted from backend development, and if you want multiple interfaces (like native mobile and web), they both talk directly to the same backend. Also, from the beginning you have the option to open up your application with an API for other developers to take your application new places, and, shameless plug here, add your API to Mashape to spread your new app!

eCloudEdit Part 2: CouchDB

In my last post I showed the Webmachine backend to James Yu's CloudEdit app in Backbone.js. What was left out was, where are the documents stored? Here I'll show how this is done with CouchDB. And you can give the app a try at http://erlware.org:8080

First, a new Erlang app is needed that we'll call ece_db.

  
ece_db/  
├── doc  
├── ebin  
│   ├── ece_db.app  
│   └── overview.edoc  
├── include  
└── src  
    ├── ece_db.erl  
    ├── ece_db_app.erl  
    └── ece_db_sup.erl
Three modules are implemented, one that starts the app by calling the supervisor's start function, the supervisor itself that sets up a _simple_one_for_one_ and the _gen_server_ that handles the frontends requests for creating and modifying documents in CouchDB. We'll ignore the _ece_db_app.erl_ module for this post and start with _ece_db_sup.erl_. On requests from the Webmachine resource module we don't want one to wait on the other, requests should be handled in parrallel. One option is to not create a process for the database backend and instead have all of the functions in the database interface run in the process of the Webmachine resource. However, two reasons to not do this are that there are multiple pieces of data that must be read out of a configuration file and used to setup what is needed to talk to CouchDB. We do not want to be doing this for every request! Furthermore a supervised _gen_server_ allows us the possibility of retrying requests with no extra code and crashing without retrying but not taking down the resource's process that is handling the user's HTTP request. It is a lot nicer and easier to just let things fail when we can!
  
  {ece_db, [{server, "HOSTNAME"},  
            {port, 80},  
            {database, "ecloudedit"}]}
  
-spec init(list()) -> {ok, {SupFlags::any(), [ChildSpec::any()]}} |  
                       ignore | {error, Reason::any()}.  
init([]) ->  
    RestartStrategy = simple_one_for_one,  
    MaxRestarts = 0,  
    MaxSecondsBetweenRestarts = 1,  

    SupFlags = {RestartStrategy, MaxRestarts, MaxSecondsBetweenRestarts},  

    Restart = temporary,  
    Shutdown = 2000,  
    Type = worker,  

    {ok, Server} = application:get_env(server),  
    {ok, Port} = application:get_env(port),  
    {ok, DB} = application:get_env(database),  

    AChild = {ece_db, {ece_db, start_link, [Server, Port, DB]},  
              Restart, Shutdown, Type, [ece_db]},  

    {ok, {SupFlags, [AChild]}}.
First, we have the config entries in config/sys.config that will be read in by _application:gen_env_. Here we set the server URL, port number and name of the database to use. CouchDB can easily be run locally but for my running copy I use a database hosted by the great service [Cloudant](http://www.cloudant.com). Next is the _init_ function for _ece_db_sup.erl_. A key thing to note for a _simple_one_for_one_ is that NO process is started after the supervisor's init function returns like with other types of supervisor children. Instead we must explicitly call _start_child_. This is how we are able to create a supervised _gen_server_ process for every HTTP request. Below is the code in _ece_db_sup_ for starting and stopping the process:
  
start_child() ->  
    supervisor:start_child(?SERVER, []).  

terminate_child(PID) ->  
    ece_db:terminate_child(PID).
In the last post I showed the functions that start and stop the _ece_db_ process. Here they are again:
  
init([]) ->  
    {ok, PID} = ece_db_sup:start_child(),  
    {ok, #ctx{db=PID}}.
  
finish_request(ReqData, Ctx) ->  
    ece_db_sup:terminate_child(Ctx#ctx.db),  
    {true, ReqData, Ctx}.
On each request Webmachine calls the _init_ function for the resource that is matched in the dispatch table. In that init function we call _start_child_ in _ece_db_sup_ which returns _{ok, PID}_. The PID is the process id we'll be sending messages. Now in _ece_db_ we implement the API functions needed to start the process and to interact with it.
  
start_link(Server, Port, DB) ->  
    gen_server:start_link(?MODULE, [Server, Port, DB], []).  

all(PID) ->  
    gen_server:call(PID, all).  

find(PID, ID) ->  
    gen_server:call(PID, {find, ID}).  

create(PID, Doc) ->  
    gen_server:call(PID, {create, Doc}).  

update(PID, ID, JsonDoc) ->  
    gen_server:call(PID, {update, ID, JsonDoc}).  

terminate(PID) ->  
    gen_server:call(PID, terminate).  

Each function, besides the one for starting the process, takes a PID. This PID is the process genserver:call_ to which it sends its message. To handle these messages we have the genserver_ callbacks.

  
%%%===================================================================  
%%% gen_server callbacks  
%%%===================================================================  
  
%% @private  
init([Server, Port, DB]) ->  
    CouchServer = couchbeam:server_connection(Server, Port, "", []),  
    {ok, CouchDB} = couchbeam:open_db(CouchServer, DB),  
    {ok, #state{db=CouchDB}}.  

%% @private  
handle_call(all, _From, #state{db=DB}=State) ->  
    Docs = get_docs(DB, [{descending, true}]),  
    {reply, mochijson2:encode(Docs), State};  
handle_call({find, ID}, _From, #state{db=DB}=State) ->  
    [Doc] = get_docs(DB, [{key, list_to_binary(ID)}]),  
    {reply, mochijson2:encode(Doc), State};  
handle_call({create, Doc}, _From, #state{db=DB}=State) ->  
    {ok, Doc1} = couchbeam:save_doc(DB, Doc),  
    {NewDoc} = couchbeam_doc:set_value(<<"id">>, couchbeam_doc:get_id(Doc1), Doc1),  
    {reply, mochijson2:encode(NewDoc), State};  
handle_call({update, ID, NewDoc}, _From, #state{db=DB}=State) ->  
    IDBinary = list_to_binary(ID),  
    {ok, Doc} = couchbeam:open_doc(DB, IDBinary),  
    NewDoc2 = couchbeam_doc:set_value(<<"_id">>, IDBinary, {NewDoc}),  
    NewDoc3 = couchbeam_doc:set_value(<<"_rev">>, couchbeam_doc:get_rev(Doc), NewDoc2),  
    {ok, {Doc1}} = couchbeam:save_doc(DB, NewDoc3),  
    {reply, mochijson2:encode(Doc1), State;  
handle_call(terminate, _From, State) ->  
    {stop, normal, State}.

init takes the server URL, the port number and the name of the database to store the documents as arguments and creates a CouchBeam database record. To handle the other messages we have the handlecall_ function to match on the different tuples sent to genserver:call_ for the different function calls. all uses the internal function getdocs_ with the option to have the function in descending order (so the newest is first) and returns those after encoding to JSON. find uses the same function but only wants a certain document which in the case of CouchDB is specified with a key value. create first saves the new document which returns the document that is actually stored in CouchDB. CouchDB adds the _id and _rev key/value pairs it generates. Since our frontend expects the id to be the value of a key id and not _id we use couchbeamdoc:setvalue to set a value of the new key id with the value of the documents id retrieved from the document with couchbeamdoc:getid. For update we first open the document corresponding to the id passed in as an argument to get the _rev value. _rev needs to be set in the document we send to savedoc_ so CouchDB knows this is a valid update based on the previous version of the document. Thus we set the new documents _id and _rev values and send to savedoc_.

Lastly, we have the getdocs_ function. Since the frontend expects the key id to exist and CouchDB stores the id with the key _id we use a simple view to return the documents with id containing the value of the documents _id. Views are defined with map/reduce functions that CouchDB runs across the databases documents and builds an index from. These indexes are then queried based on keys to find documents. Thus access is very quick, indexes are built with BTrees, and a map/reduce does not have to be run on the documents for every request. Additionally, when a new document is created the view's map/reduce functions only need to run over the new documents to update the BTree. Updates to the view's indexes are either done on each use of the view or can be manually told to run.

  
function (doc)  
{  
  emit(doc._id, {id : doc._id, title : doc.title, created_at : doc.created_at, body : doc.body});  
}
  
get_docs(DB, Options) ->  
    {ok, AllDocs} = couchbeam:view(DB, {"all", "find"}, Options),  
    {ok, Results} = couchbeam_view:fetch(AllDocs),  

    {[{<<"total_rows">>, _Total},  
      {<<"offset">>, _Offset},  
      {<<"rows">>, Rows}]} = Results,  

    lists:map(fun({Row}) ->  
                      {<<"value">>, {Value}} = lists:keyfind(<<"value">>, 1, Row),  
                      Value  
              end, Rows).  

In getdocs_ the couchbeam:view function is used to construct a view request to be sent to the CouchDB server. couchbeamview:fetch_ sends the view request and returns the results as Erlang terms. After this we extract the rows from the results and use lists:map to extract out just the value of each row to be returned.

That's it! There are places that could use improvement. One being the use of lists:map over every returned row to construct documents the frontend can deal with. Additionally, the need to duplicate _id as id for the frontends use could be removed through modifications on the frontend.

In the next installment I'll be updating the code -- and maybe making some of these performance enhancements -- with James Yu's recent changes in his Part 2.

eCloudEdit: Erlang, Webmachine and Backbone.js

To experiment with using a pure client-side rendering talking to an Erlang backend I've taken James Yu's CloudEdit tutorial an app written with Backbone.js and Rails and converted the backend to use Webmachine and CouchDB. You can see eCloudEdit in action here. The Backbone.js code is the same so to understand that please see James' post, here I'll describe the Erlang backend.

To begin with we setup two applications, one for handling the web interaction and a second for handling the database interaction. We end up with this directory layout:

  |-eCloudEdit  
   |-bin  
   |-config  
   |-doc  
   |-lib  
   |---ece_db  
   |-----doc  
   |-----ebin  
   |-----include  
   |-----src  
   |---ece_web  
   |-----doc  
   |-----ebin  
   |-----include  
   |-----priv  
   |-------images  
   |-------javascripts  
   |---------controllers  
   |---------models  
   |---------views  
   |-------stylesheets  
   |-----src

Under eceweb/priv is where all the html and Javascript from CloudEdit is placed, with nothing changed. To serve up the static content through Webmachine we use this resource which we won't worry about in this article. Lets look at how webmachine deciced where to route requests. This is handled by a set of dispatch rules that are passed into Webmachine on startup. We can see how this is done by looking at eceweb_sup and how the supervisor loads Webmachine:

[

-spec init(list()) -> {ok, {SupFlags::any(), [ChildSpec::any()]}} |  
                          ignore | {error, Reason::any()}.  
init([]) ->  
    WebChild = {webmachine_mochiweb,  
                {webmachine_mochiweb, start, [config()]},  
                permanent, 5000, worker, dynamic},  

    RestartStrategy = one_for_one,  
    MaxRestarts = 3,  
    MaxSecondsBetweenRestarts = 10,  
    SupFlags = {RestartStrategy, MaxRestarts, MaxSecondsBetweenRestarts},  

    {ok, {SupFlags , [WebChild]}}.  

config() ->  
    {ok, IP} = application:get_env(webmachine_ip),  
    {ok, Port} = application:get_env(webmachine_port),  
    {ok, App}= application:get_application(),  
    LogDir = code:priv_dir(App) ++ "/logs",  
    {ok, Dispatch} = file:consult(filename:join([code:priv_dir(App), "dispatch"])),  

    [{ip, IP},  
     {port, Port},  
     {log_dir, LogDir},  
     {backlog, 128},  
     {dispatch, Dispatch}].

](https://github.com/tsloughter/eCloudEdit/blob/master/lib/eceweb/src/eceweb_sup.erl)

The dispatch terms are loaded from a file dispatch in the application's priv directory. For this app the dispatch file contains:

[

{["documents", id], ece_resource_documents, []}.  
{["documents"], ece_resource_documents, []}.  
{['*'], ece_resource_static, []}.

](https://github.com/tsloughter/eCloudEdit/blob/master/lib/ece_web/priv/dispatch)

There are two resources, one for handling the requests for creating, updating and viewing documents and one for serving up all other requests (static html, js and css files). The first rule matches paths like, /documents/foo, id is set to foo and the request is sent to the documents resource. If there is nothing after /documents it is still sent to the documents resource but there is no id.

Webmachine is essentially a REST toolkit. You build resources by defining functions that handle the different possible HTTP requests. For this webapp we'll only be dealing with GET, POST and PUT. GET is used if we'd like to retrieve information on documents, POST is for creating a new document and PUT is for updating a document.

[

-record(ctx, {db}).  

init([]) ->  
    {ok, PID} = ece_db_sup:start_child(),  
    {ok, #ctx{db=PID}}.  

allowed_methods(ReqData, Ctx) ->  
    {['HEAD', 'GET', 'POST', 'PUT'], ReqData, Ctx}.  

content_types_accepted(ReqData, Ctx) ->  
    {[{"application/json", from_json}], ReqData, Ctx}.  

content_types_provided(ReqData, Ctx) ->  
    {[{"application/json", to_json}], ReqData, Ctx}.  

finish_request(ReqData, Ctx) ->  
    ece_db_sup:terminate_child(Ctx#ctx.db),  
    {true, ReqData, Ctx}.

](https://github.com/tsloughter/eCloudEdit/blob/master/lib/eceweb/src/eceresource_documents.erl)

To handle GET requests we implemented the tojson_ function we specified in contenttypesprovided to handle GET requests for json data.

[

to_json(ReqData, Ctx) ->  
    case wrq:path_info(id, ReqData) of  
        undefined ->  
            All = ece_db:all(Ctx#ctx.db),  
            {All, ReqData, Ctx};  
        ID ->  
            JsonDoc = ece_db:find(Ctx#ctx.db, ID),  
            {JsonDoc, ReqData, Ctx}  
    end.

](https://github.com/tsloughter/eCloudEdit/blob/master/lib/eceweb/src/eceresource_documents.erl)

wrq:pathinfo_ is used to find the value of id, which we set in the dispatch file, if it is undefined we know the request is for all documents, while if it has a value we know to find the contents of the document with that id and return its contents. We'll see the content of ecedb:all/1_ and ecedb:find/2_ in the next article. Just know they both return JSON data structures.

Now we must support POST for creating documents or we have nothing to return to a GET request.

[

process_post(ReqData, Ctx) ->  
    [{JsonDoc, _}] = mochiweb_util:parse_qs(wrq:req_body(ReqData)),  
    {struct, Doc} = mochijson2:decode(JsonDoc),  
    NewDoc = ece_db:create(Ctx#ctx.db, {Doc}),  
    ReqData2 = wrq:set_resp_body(NewDoc, ReqData),  
    {true, ReqData2, Ctx}.

](https://github.com/tsloughter/eCloudEdit/blob/master/lib/eceweb/src/eceresource_documents.erl)

wrq:reqbody/1_ returns the contents of the body sent in the HTTP request. Here it is the conents of the document to store. We decode it to an Erlang data structure and pass it to the ecedb_ app for inserting into the database. After inserting to the database the create/2 function returns the new document with a new element id (in this case generated by CouchDB). This is required so we know the document's id which is used by the Backbone.js frontend. In order to return it from the POST request we must set response body to the contents of the document with wrq:setrespbody/2

Lastly, updating documents requires support for PUT. In contentstypeaccepted/2 we've specified that PUT requests with JSON content is sent to the function fromjson/2_:

[

from_json(ReqData, Ctx) ->  
    case wrq:path_info(id, ReqData) of  
        undefined ->  
            {false, ReqData, Ctx};  
        ID ->  
            JsonDoc = wrq:req_body(ReqData),  
            {struct, Doc} = mochijson2:decode(JsonDoc),  
            NewDoc = ece_db:update(Ctx#ctx.db, ID, Doc),  
            ReqData2 = wrq:set_resp_body(NewDoc, ReqData),  
            {true, ReqData2, Ctx}  
    end.

](https://github.com/tsloughter/eCloudEdit/blob/master/lib/eceweb/src/eceresource_documents.erl)

If this request was not routed through the first rule in our dispatch file it does not have an id and thus can not be an update. When this happens we return false so the frontend is aware something has gone wrong. For requests containing an id we pass the contents of the requests body to ecedb's _update/2 function.

In the next post I'll show how ecedb_ is implemented with Couchbeam for reading and writing the documents to CouchDB on Cloudant.