Getting Flymake and Rebar to Play Nice

TLDR;
Copy and paste the following into your elisp erlang-mode configuration to get flymake working with Rebar projects.

  
 (defun ebm-find-rebar-top-recr (dirname)  
      (let* ((project-dir (locate-dominating-file dirname "rebar.config")))  
        (if project-dir  
            (let* ((parent-dir (file-name-directory (directory-file-name project-dir)))  
                   (top-project-dir (if (and parent-dir (not (string= parent-dir "/")))  
                                       (ebm-find-rebar-top-recr parent-dir)  
                                      nil)))  
              (if top-project-dir  
                  top-project-dir  
                project-dir))  
              project-dir)))  

    (defun ebm-find-rebar-top ()  
      (interactive)  
      (let* ((dirname (file-name-directory (buffer-file-name)))  
             (project-dir (ebm-find-rebar-top-recr dirname)))  
        (if project-dir  
            project-dir  
          (erlang-flymake-get-app-dir))))  

     (defun ebm-directory-dirs (dir name)  
        "Find all directories in DIR."  
        (unless (file-directory-p dir)  
          (error "Not a directory `%s'" dir))  
        (let ((dir (directory-file-name dir))  
              (dirs '())  
              (files (directory-files dir nil nil t)))  
            (dolist (file files)  
              (unless (member file '("." ".."))  
                (let ((absolute-path (expand-file-name (concat dir "/" file))))  
                  (when (file-directory-p absolute-path)  
                    (if (string= file name)  
                        (setq dirs (append (cons absolute-path  
                                                 (ebm-directory-dirs absolute-path name))  
                                           dirs))  
                        (setq dirs (append  
                                    (ebm-directory-dirs absolute-path name)  
                                    dirs)))))))  
              dirs))  

    (defun ebm-get-deps-code-path-dirs ()  
        (ebm-directory-dirs (ebm-find-rebar-top) "ebin"))  

    (defun ebm-get-deps-include-dirs ()  
       (ebm-directory-dirs (ebm-find-rebar-top) "include"))  

    (fset 'erlang-flymake-get-code-path-dirs 'ebm-get-deps-code-path-dirs)  
    (fset 'erlang-flymake-get-include-dirs-function 'ebm-get-deps-include-dirs)  

Intro

Its probably no great surprise to anyone that I dislike Rebar a lot. That said there are times when I have no choice but to use it. This is always either because a company I am contracting for uses it, or an open source project I am contributing to uses it. When I am forced to use it there are a few things I don't want to give up. Most important among these is Flymake for Erlang. The default setup for Flymake doesn't work for Rebar projects because Flymake does not know where the code and include paths for dependencies are. Fortunately, we can fix this with a few lines of elisp.

Flymake For Erlang

First make sure you have Flymake for Erlang installed. It is easiest just to follow the directions available on the Erlang Website.

The Elisp Additions for Erlang Flymake

There are two defvars that point to functions that are used to search for the correct code paths and include paths respectively. We are going to replace those functions with our own functions. Both these functions search upwards from the directory that contains the file pointed to by the current buffer, looking for the top most 'rebar.config' in the directory path. It then uses that for a base and searches down the directory structure looking for either 'ebin' files or 'include' files.

There are two things to note here. The first is that you must have already run get-deps for rebar for this to work and the second is that if your project is truly huge or you have way more dependencies then you probably need this search could take a second or two. That is a second or two too long in an interactive compiler like Flymake. That said, the likelihood that you will run into this second problem is quite low.

Getting Started

The very thing you want to do is ensure that you have required the erlang-flymake module. Most of what we do below depends on this.

  
(require 'erlang-flymake)  

Finding the Top rebar.config

The second thing we want to do is look for the top rebar.config in the project. If a rebar project contains more then one OTP application its quite likely that it will contain more then one rebar.config. The very topmost rebarconfig` is the right one to serve as root of our search. So we introduce a set of recursive functions to look for that top level dir.

  
    (defun ebm-find-rebar-top-recr (dirname)  
      (let* ((project-dir (locate-dominating-file dirname "rebar.config")))  
        (if project-dir  
            (let* ((parent-dir (file-name-directory (directory-file-name project-dir)))  
                   (top-project-dir (if (and parent-dir (not (string= parent-dir "/")))  
                                       (ebm-find-rebar-top-recr parent-dir)  
                                      nil)))  
              (if top-project-dir  
                  top-project-dir  
                project-dir))  
              project-dir)))  

ebm-find-rebar-top-recr will return either the top most directory or nil. Our next function takes that result and does something useful with.

  
    (defun ebm-find-rebar-top ()  
      (interactive)  
      (let* ((dirname (file-name-directory (buffer-file-name)))  
             (project-dir (ebm-find-rebar-top-recr dirname)))  
        (if project-dir  
            project-dir  
          (erlang-flymake-get-app-dir))))  

In this function, we get the directory containing the file pointed at by the current buffer. We then call our recr function. If it returns a directory we return that, if it returns nil however, we call the original erlang-flymake-get-app-dir function.

At this point we should have our project root. Now its a simple matter of recursively searching down the directory tree looking for files of a certain name. So we create a function that does just that, given a directory and a name will return a list of absolute paths for each subdirectory that matches the specified name.

  
(defun ebm-directory-dirs (dir name)  
        "Find all directories in DIR."  
        (unless (file-directory-p dir)  
          (error "Not a directory `%s'" dir))  
        (let ((dir (directory-file-name dir))  
              (dirs '())  
              (files (directory-files dir nil nil t)))  
            (dolist (file files)  
              (unless (member file '("." ".."))  
                (let ((absolute-path (expand-file-name (concat dir "/" file))))  
                  (when (file-directory-p absolute-path)  
                    (if (string= file name)  
                        (setq dirs (append (cons absolute-path  
                                                 (ebm-directory-dirs absolute-path name))  
                                           dirs))  
                        (setq dirs (append  
                                    (ebm-directory-dirs absolute-path name)  
                                    dirs)))))))  
              dirs))  

Now we write a couple of functions to replace the corresponding functions in erlang-flymake. The first looks for all ebin directories while the second looks for all include directories.

  
    (defun ebm-get-deps-code-path-dirs ()  
        (ebm-directory-dirs (ebm-find-rebar-top) "ebin"))  

    (defun ebm-get-deps-include-dirs ()  
       (ebm-directory-dirs (ebm-find-rebar-top) "include"))  

Finally we replace the erlang-flymake versions of those functions with our implementations.

  
(fset 'erlang-flymake-get-code-path-dirs 'ebm-get-deps-code-path-dirs)  
(fset 'erlang-flymake-get-include-dirs-function 'ebm-get-deps-include-dirs)  

Conclusion

This approach is a bit of a hack, we basically use some heuristics to find a root and then just grab everything under that that looks remotely like a code or include directory. While its a bit hacky it has the valuable upside that its flexible and robust.

Erlang Common Test Continuous Integration

Common Test is a well thought out integration testing framework for Erlang. If you
are not using it you probably should be. However, it has one fault. It
does not return non-negative exit status' to the caller when the tests
fail. This is a major oversight, and it makes it difficult to use as
part of a continuous integration scheme or in a make based build
system.

The long term fix is for the OTP folks to resolve the issue in the
ct_run command. To that end I have filed a bug report with the
Erlang folks. In the short term, though, we need this behaving
correctly. After much twiddling around with different solutions and
conversions on the erlang-questions list. This solution finally popped
out of a conversation with Lukas Larsson. Basically, we use the old
unix standby of awk.

  

    ct_run -dir tests  ... | awk "/FAILED/{exit 1;}/failed/{exit 1;}/SKIPPED/{exit 1;}"  

Where ... is replaced with your additional options. Its not the best
solution on the planet, but it is the simplest one that I found that
works consistently.

Sinan Releases and Being Right

Fred, of Learn You Some Erlang for Great Good, today posted on his blog about the problems around how rebar handles releases, Rebar Releases and Being Wrong. The problems he mentions and a few others are why, despite giving it a legitimate shot, I have found rebar unusable for my workflow to be efficient and stable while adhering to OTP standards at the same time.

I suggest first reading his post, if you already use rebar, and then continuing on with the rest of this.

I'll start with an example on the generation of a project containing two applications and a dependency from one of those applications of cowboy. Next, I'll create a release (and in the process a deployable target system) to show the difference in how sinan handles this process.

TL;DR Sinan does OTP the right way, rebar does not.

First, you can download the latest version sinan from this link, it is simply an executable escript, so 'chmod +x sinan' and put it in your PATH and you are good to go.

Sinan provides a 'gen' command to create your project. I include the output of the steps I took to build this project. Sinan assumes this is a multiple application project, but if you give "y" instead it will create a directory structure similar to rebars default structure with a src/ directory instead of a lib/ directory.

$ sinan gen  
Please specify your name  
your name> Tristan Sloughter  
Please specify your email address  
your email> [email protected]  
Please specify the copyright holder  
copyright holder ("Tristan Sloughter")>  
Please specify name of your project  
project name> rel_example  
Please specify version of your project  
project version> 0.0.1  
Please specify the ERTS version ("5.9")>  
Is this a single application project ("n")>  
Please specify the names of the OTP apps that will be developed under this project. One application to a line. Finish with a blank line.  
app> app_1  
app ("")> app_2  
app ("")>  
Would you like a build config? ("y")> y  
Project was created, you should be good to go!

We now have a project named rel_example and can see the generated contents.

$ cd rel_example/  
$ ls  
config  doc  lib  sinan.config

Before going further I add the line {include_erts, true}. to sinan.config so that a generated tarball of the release contains erts and can be booted on a machine without Erlang installed.

$ cat sinan.config  
{project_name, rel_example}.  
{project_vsn, "0.0.1"}.  

{build_dir,  "_build"}.  

{ignore_dirs, ["_", "."]}.  

{ignore_apps, []}.  

{include_erts, true}.

A tree structure view of the generated project is below:

.  
├── config  
│   └── sys.config  
├── doc  
├── lib  
│   ├── app_1  
│   │   ├── doc  
│   │   ├── ebin  
│   │   │   └── overview.edoc  
│   │   ├── include  
│   │   └── src  
│   │       ├── app_1_app.erl  
│   │       ├── app_1.app.src  
│   │       └── app_1_sup.erl  
│   └── app_2  
│       ├── doc  
│       ├── ebin  
│       │   └── overview.edoc  
│       ├── include  
│       └── src  
│           ├── app_2_app.erl  
│           ├── app_2.app.src  
│           └── app_2_sup.erl  
└── sinan.config

You'll see we have a lib directory with two applications containing their source files under a src directory. Now in order to boot the release we'll create, we need to remove a couple tings from each supervisor. Instead of creating something for them to supervise just remove the variable AChild and replace [AChild] with [].

Next, so we have a third party dependency in the example, add cowboy to the applications in nano lib/app1/src/app1.app.src:

{applications, [kernel, stdlib, cowboy]},

Sinan provides a depends command to show the depenedencies of the project and where they are located:

$ sinan depends -v  
starting: depends  
Using the following lib directories to show resolved dependencies and where it found them:  

    /home/tristan/.kerl/installs/r15b/lib  
    /home/tristan/Devel/rel_example/_build/rel_example/lib  

compile time dependencies:  

runtime dependencies:  

    kernel                    2.15       : /home/tristan/.kerl/installs/r15b/lib/kernel-2.15  
    stdlib                    1.18       : /home/tristan/.kerl/installs/r15b/lib/stdlib-1.18  
    cowboy                    0.5.0      : /home/tristan/.kerl/installs/r15b/lib/cowboy-0.5.0  

project applications:  

    app_1                     0.1.0      : /home/tristan/Devel/rel_example/_build/rel_example/lib/app_1-0.1.0  
    app_2                     0.1.0      : /home/tristan/Devel/rel_example/_build/rel_example/lib/app_2-0.1.0

Now lets build a release and target system.

$ sinan dist

After running the dist command we have a _build directory that we find the following structure. I removed the files/dirs under each app to shorten the listing.

_build/  
├── rel_example  
│   ├── bin  
│   │   ├── rel_example  
│   │   └── rel_example-0.0.1  
│   ├── erts-5.9  
│   │   ├──   
│   ├── lib  
│   │   ├── app_1-0.1.0  
│   │   │   ├──   
│   │   ├── app_2-0.1.0  
│   │   │   ├──   
│   │   ├── cowboy-0.5.0  
│   │   │   ├──   
│   │   ├── kernel-2.15  
│   │   │   ├──   
│   │   └── stdlib-1.18  
│   │       ├──   
│   └── releases  
│       └── 0.0.1  
│           ├── rel_example.boot  
│           ├── rel_example.rel  
│           ├── rel_example.script  
│           └── sys.config  
└── tar  
    └── rel_example-0.0.1.tar.gz

Sinan has created a lib directory containing all necessary applications for our release as well as the needed files for booting the release. Additionally the dist command creates a tar.gz for easy deployment. But if we simply want to run our release where we are we can:

$ _build/rel_example/bin/rel_example  
Erlang R15B (erts-5.9)  

[64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]  

Eshell V5.9  (abort with ^G)  
1>

This is only the tip of the iceberg of what sinan is capable of. I can't go into all of it here but I'll mention that you are able to define multiple releases for a project to generate and which of your project apps to include in each. Additionally you are able to provide a custom rel file if you require tweaks.

The important part to take away from this post is the structure of what you are working with when using sinan and how it is based on OTP standards, both for the source you work on and the results of the build process under _build/.

Cowboy and Batman.js for Erlang Web Development

Why Cowboy and Batman.js

There are a lot of Erlang web frameworks out there today. Not all are modeled after the MVC model (see Nitrogen), but I think all of them are addressing the problem the wrong way. I recently gave a presentation, slides here and the code for this example here, describing my perferred method for using Erlang for web development and why I think it is the best way to go. In this post, I'll go into more details on how to build the Erlang backend for the TodoMVC clone I did with Batman.js. I will not spend time on Batman.js but instead only give a quick list of reasons I prefer it to other Javascript frameworks.

Batman.js advantages:

  • Automatic URL generation based on model
  • HTML data-bind templates
  • Coffeescript
    Cowboy is a newer Erlang web server that provides a REST handler based on Webmachine. Both of these are perfect for developing a RESTful API, because they follow the HTTP standard exactly and when you are building an API based on HTTP, being able to properly reason about how the logic of the application maps to the protocol eases development and eases getting REST "right".

Nginx

Any non-dynamic content should be served by Nginx since there is no logic needed and it is something Nginx is great at, so why have Erlang do it? The snippet below configures Nginx to listen on port 80 and serve files from bcmvc's priv directory. Each request is checked to see if it is a POST or any other method with a JSON request type. If either of those are true, the request is proxied on to a server listening on port 8080, in our case the Cowboy server.

server {  
  listen 80;  
  server_name localhost;  

  location / {  
    root   <PATH TO CLONE>/bcmvc/lib/bcmvc_web/priv/;  

    if ($request_method ~* POST) {  
      proxy_pass        http://localhost:8080;     }  

    if ($http_accept ~* application/json) {  
      proxy_pass        http://localhost:8080;     }  
  }  
}

The API

Batman.js knows what endpoints to use and what data to send based on the name of the model we created and the encoded variables, code here. This results in the following API:

 
Method Endpoint Data Return
POST _todos_ {todo : {body:"bane wants to meet, not worried",isDone:false}}
PUT /todos/33e93b30-2371-4071-afc5-2d48226d5dba {todo : {body:"bane wants to meet, not worried",isDone:false}}
GET _todos_ [{todo : {id:"33e93b30-2371-4071-afc5-2d48226d5dba", body:"bane wants to meet, not worried", isDone:false}}]
DELETE /todos/33e93b30-2371-4071-afc5-2d48226d5dba

Cowboy Dispatch and Supervisor

Dispatch rules are matched by Cowboy to know what handler to send the request to. Here we have two rules. One that matches just the URL /todos and one that matches the URL with an additional element which will be associated with the atom todo. Both requests will be sent to the module bcmvctodohandler.

Dispatch = [{'_', [{[<<"todos">>], bcmvc_todo_handler, []},  
                   {[<<"todos">>, todo], bcmvc_todo_handler, []}]}],

Cowboy provides a useful function childspec_ for creating a child specfication to use in our supervisor. The child spec here tells Cowboy we want a TCP listener on port 8080 that handles the HTTP protocol. We additionally provide our dispatch list for it to match against and pass on requests.

ChildSpec = cowboy:child_spec(bcmvc_cowboy, 100, cowboy_tcp_transport,  
                              [{port, 8080}], cowboy_http_protocol, [{dispatch, Dispatch}]),

Cowboy Handler

Now that we have a server on port 8080 that knows to send certain requests to our todo handler, we can build the module. The first required function to export is init/3. This function let's Cowboy know we have a REST protocol, this is how it knows what functions to call (some have defaults and some existing in our module) to handle the request.

init(_Transport, _Req, _Opts) ->  
    {upgrade, protocol, cowboy_http_rest}.

Knowing that this is a REST handler Cowboy will pass the request on to allowedmethods/2_ to find out if our handler is able to handle this method. Next, the content types accepted and provided by the handler are checked against the incoming request. The expected HTTP response status codes are returned if any of these fail. 405 for allowedmethods, XXX for _contenttypesaccepted and XXX for contenttypesprovided.

allowed_methods(Req, State) ->  
    {['HEAD', 'GET', 'PUT', 'POST', 'DELETE'], Req, State}.  

content_types_accepted(Req, State) ->  
    {[{{<<"application">>, <<"json">>, []}, put_json}], Req, State}.  

content_types_provided(Req, State) ->  
    {[{{<<"application">>, <<"json">>, []}, get_json}], Req, State}.

Now the request is sent to the function that handles the HTTP method type of the request.

For a POST, a request to create a new todo item, the function processpost/2_ is sent the request. Here we retrieve the body, a JSON object, from the request, convert it to a record and save the model. We'll see how this record conversion is done when we look at the model module. To inform the frontend of the id of our new resource we set the location header to be the path with the id.

process_post(Req, State) ->  
    {ok, Body, Req1} = cowboy_http_req:body(Req),  
    Todo = bcmvc_model_todo:to_record(Body),  
    bcmvc_model_todo:save(Todo),  

    NewId = bcmvc_model_todo:get(id, Todo),  
    {ok, Req2} = cowboy_http_req:set_resp_header(  
                   <<"Location">>, <<"/todos/", NewId/binary>>, Req1),  

    {true, Req2, State}.

For this handler we expect PUT for an update to an object, that is what Batman.js does, but a PATCH would make more sense. For a PUT the URL contains the id for the todo item to be updated. That is retrieved with the_ binding/2_ function. The todo record is created the same as in processpost/2_ but then the this id is set for the model and the update/1 function is used to save it to the database.

put_json(Req, State) ->  
    {ok, Body, Req1} = cowboy_http_req:body(Req),  
    {TodoId, Req2} = cowboy_http_req:binding(todo, Req1),  
    Todo = bcmvc_model_todo:to_record(Body),  
    Todo2 = bcmvc_model_todo:set([{id, TodoId}], Todo),  
    bcmvc_model_todo:update(Todo2),      
    {true, Req2, State}.

For a GET request, which for this application we do not deal with a request for a single todo item, all todo items are retrieved from the model module. Each of these is passed to the model's tojson/1_ function and the result of converting each to JSON is combined into a binary string and placed between brackets so the Batman.js frontend receives a proper JSON list of JSON objects.

get_json(Req, State) ->  
    JsonModels = lists:foldr(fun(X, <<"">>) ->  
                                 X;  
                            (X, Acc) ->  
                                 <<Acc/binary, ",", X/binary>>  
                         end, <<"">>, [bcmvc_model_todo:to_json(Model) || Model <- bcmvc_model_todo:all()]),  

    {<<"[", JsonModels/binary, "]">>, Req, State}.

And lastly, DELETE. Like in PUT the todo item's id is retrieved from the bindings created based on the dispatch rules and this is passed to the model's delete function.

delete_resource(Req, State) ->  
    {TodoId, Req1} = cowboy_http_req:binding(todo, Req),  
    bcmvc_model_todo:delete(TodoId),  
    {true, Req1, State}.

Models

Model's are repsented as records and must provide serialization functions to go between JSON and a record. Each model uses a parse transform that creates functions for creating and updating the record. The transform is a modified version of exprecs from Ulf Wiger that also uses the type definitions in the record to ensure when setting a field that it is the correct type. For example in the todo model isDone is a boolean, so when the model is created the boolean convert function will be matched to convert the string representation to an atom:

convert(boolean, <<"false">>) ->  
    false;  
convert(boolean, <<"true">>) ->  
    true;

So the key pieces of the_ bcmvcmodeltodo_ are:

-compile({parse_transform, bcmvc_model_transform}).  

-record(bcmvc_model_todo, {id = ossp_uuid:make(v1, text) :: string(),  
                           body                          :: binary(),  
                           isDone                        :: boolean()}).  

to_json(Record) ->  
    ?record_to_json(?MODULE, Record).  

to_record(JSON) ->  
    ?json_to_record(?MODULE, JSON).

The_ ?recordtojson_ and ?jsontorecord macros are defined in jsonerl.hrl. These marcos are generic and work for any record that is typed and uses the model transform.

Conclusion

Clearly, much of what the resource handler and model do is generic and can be abstracted out so that implementing new models and resources can be even simpler. This is the goal of my project Maru. Currently it is based on Webmachine but is now being convered to Cowboy.

In the end, using Cowboy for building a RESTful interface for your application allows you to build interfaces for the frontend entirely separted from backend development, and if you want multiple interfaces (like native mobile and web), they both talk directly to the same backend. Also, from the beginning you have the option to open up your application with an API for other developers to take your application new places, and, shameless plug here, add your API to Mashape to spread your new app!

Erlang, Cowboy and Batman.js for Building Web Applications

I'll have a complete walk of through using Cowboy and Batman.js to build the TodoMVC clone in a few days. For now I have the slides from my talk at the Chicago Erlang User Group:

Chicago Erlang User Group April, 4th 2012

I couldn't get iframe embedding to work with Wordpress... So if anyone knows what that is up with please comment.