Here is your chance to get our book Erlang and OTP in Action for half price on April 16th. Use code dotd0416au at www.manning.com/logan/
TL;DR
As I’ve mentioned before, Opa is a new web framework that introduces not only the framework itself but a whole new language. A lot has changed in Opa since I last posted about it. Now Opa has a Javascript-esque look and runs on Node.js. But it still has the amazing typing system that makes Opa a joy to code in.
The currently available Heroku buildpack for Opa only supported the old, pre-Node, support. So I’ve created an all new buildpack and here I will show both a bit of how I created that buildpack and how to use it to run your Opa apps on Heroku.
The first step was creating a tarball of Opa that would work on Heroku. For this I used the build tool vulcan. Vulcan is able to build software on Heroku in order to assure what is built will work on Heroku through your buildpacks.
vulcan build -v -s ./opalang/ -c "mkdir /app/mlstate-opa && yes '' | ./opa-1.0.7.x64.run" -p /app/mlstate-opa
This command is telling vulcan to build what is in the directory opalang with a command that creates the directory /app/mlstate-opa and then runs the Opa provided install script to unpack the system. This is much simpler than building Opa from source, but it is still necessary to still use vulcan to create the tarball from the output of the install script to ensure paths are correct in the Opa generated scripts.
After this run, by vulcan’s default, we will have /tmp/opalang.tgz. I upload this to S3, so that our buildpack is able to retrieve it.
Since Opa now relies on Node.js, the new buildpack must install both Node.js and the opalang.tgz that was generated. To do this I simply copied from the Node.js buildpack.
If you look at the Opa buildpack you’ll see, as with any buildpack, it consists of three main scripts under ./bin/: compile, detect and release. There are three important parts for understanding how your Opa app must be changed to be supported by the buildpack.
First, the detect script relies on there being a opa.conf to detect this being an Opa application. This for now is less important since we will be specifying the buildpack to use to the heroku script. Second, in the compile script we rely on there being a Makefile in your application for building. There is no support for simply running opa to compile the code in your tree at this time. Third, since Opa relies on Node.js and Node modules from npm you must provide a package.json file that the compile script uses to install the necessary modules.
To demostrate this I converted Opa’s hello_chat example to work on Heroku, see it on Github here.
There are two necessary changes. One, add the Procfile. A Procfile define the processes required for your application and how to run them. For hello_chat we have:
web: ./hello_chat.exe --http-port $PORT
This tell Heroku that our web process is run from the binary hello_chat.exe. We must pass the $PORT variable to the Opa binary so that it binds to the proper port that Heroku expects it to be listening on to route our traffic.
Lastly, a package.json file is added so that our buildpack’s compile script installs the necessary Node.js modules:
{
"name": "hello_chat",
"version": "0.0.1",
"dependencies": {
"mongodb" : "*",
"formidable" : "*",
"nodemailer" : "*",
"simplesmtp" : "*",
"imap" : "*"
},
"engines": {
"node": "0.8.7",
"npm": "1.1.x"
}
}
With these additions to hello_chat we are ready to create an Opa app on Heroku and push the code!
$ heroku create --stack cedar --buildpack https://github.com/tsloughter/heroku-buildpack-opa.git $ git push heroku master
The output from the push will show Node.js and npm being install, followed by Opa being unpacked and finally make being run against hello_chat. The web process in Procfile will then be run and the output will provide a link to go to our new application. I have the example running at http://mighty-garden-9304.herokuapp.com
Next time I’ll delve into database and other addon support in Heroku with Opa applications.
Working with Erlang for writing RESTful interfaces JSON is the communication “language” of choice. For simplifying the process of JSON to a model the backend could work with efficiently I’ve created maru_models. This app decodes the JSON with jiffy and uses functions generated by a modified version of Ulf’s exprecs to create an Erlang record. The generated functions are created with type information from the record definition and when a property is set for the record through these functions it is first passed to the convert function of maru_model_types to do any necessary processing.
I separated this application into a separate repo to simplify people trying the examples. But the real development will be done in the Maru main repo.
TLDR;
Copy and paste the following into your elisp erlang-mode configuration to get flymake working with Rebar projects.
(defun ebm-find-rebar-top-recr (dirname)
(let* ((project-dir (locate-dominating-file dirname "rebar.config")))
(if project-dir
(let* ((parent-dir (file-name-directory (directory-file-name project-dir)))
(top-project-dir (if (and parent-dir (not (string= parent-dir "/")))
(ebm-find-rebar-top-recr parent-dir)
nil)))
(if top-project-dir
top-project-dir
project-dir))
project-dir)))
(defun ebm-find-rebar-top ()
(interactive)
(let* ((dirname (file-name-directory (buffer-file-name)))
(project-dir (ebm-find-rebar-top-recr dirname)))
(if project-dir
project-dir
(erlang-flymake-get-app-dir))))
(defun ebm-directory-dirs (dir name)
"Find all directories in DIR."
(unless (file-directory-p dir)
(error "Not a directory `%s'" dir))
(let ((dir (directory-file-name dir))
(dirs '())
(files (directory-files dir nil nil t)))
(dolist (file files)
(unless (member file '("." ".."))
(let ((absolute-path (expand-file-name (concat dir "/" file))))
(when (file-directory-p absolute-path)
(if (string= file name)
(setq dirs (append (cons absolute-path
(ebm-directory-dirs absolute-path name))
dirs))
(setq dirs (append
(ebm-directory-dirs absolute-path name)
dirs)))))))
dirs))
(defun ebm-get-deps-code-path-dirs ()
(ebm-directory-dirs (ebm-find-rebar-top) "ebin"))
(defun ebm-get-deps-include-dirs ()
(ebm-directory-dirs (ebm-find-rebar-top) "include"))
(fset 'erlang-flymake-get-code-path-dirs 'ebm-get-deps-code-path-dirs)
(fset 'erlang-flymake-get-include-dirs-function 'ebm-get-deps-include-dirs)
Intro
Its probably no great surprise to anyone that I dislike Rebar a lot. That said there are times when I have no choice but to use it. This is always either because a company I am contracting for uses it, or an open source project I am contributing to uses it. When I am forced to use it there are a few things I don’t want to give up. Most important among these is Flymake for Erlang. The default setup for Flymake doesn’t work for Rebar projects because Flymake does not know where the code and include paths for dependencies are. Fortunately, we can fix this with a few lines of elisp.
Flymake For Erlang
First make sure you have Flymake for Erlang installed. It is easiest just to follow the directions available on the Erlang Website.
The Elisp Additions for Erlang Flymake
There are two defvars that point to functions that are used to search for the correct code paths and include paths respectively. We are going to replace those functions with our own functions. Both these functions search upwards from the directory that contains the file pointed to by the current buffer, looking for the top most ‘rebar.config’ in the directory path. It then uses that for a base and searches down the directory structure looking for either ‘ebin’ files or ‘include’ files.
There are two things to note here. The first is that you must have already run `get-deps` for rebar for this to work and the second is that if your project is truly huge or you have way more dependencies then you probably need this search could take a second or two. That is a second or two too long in an interactive compiler like Flymake. That said, the likelihood that you will run into this second problem is quite low.
Getting Started
The very thing you want to do is ensure that you have required the erlang-flymake module. Most of what we do below depends on this.
(require 'erlang-flymake)
Finding the Top rebar.config
The second thing we want to do is look for the top rebar.config in the project. If a rebar project contains more then one OTP application its quite likely that it will contain more then one rebar.config. The very topmost `rebar`config` is the right one to serve as root of our search. So we introduce a set of recursive functions to look for that top level dir.
(defun ebm-find-rebar-top-recr (dirname)
(let* ((project-dir (locate-dominating-file dirname "rebar.config")))
(if project-dir
(let* ((parent-dir (file-name-directory (directory-file-name project-dir)))
(top-project-dir (if (and parent-dir (not (string= parent-dir "/")))
(ebm-find-rebar-top-recr parent-dir)
nil)))
(if top-project-dir
top-project-dir
project-dir))
project-dir)))
ebm-find-rebar-top-recr will return either the top most directory or nil. Our next function takes that result and does something useful with.
(defun ebm-find-rebar-top ()
(interactive)
(let* ((dirname (file-name-directory (buffer-file-name)))
(project-dir (ebm-find-rebar-top-recr dirname)))
(if project-dir
project-dir
(erlang-flymake-get-app-dir))))
In this function, we get the directory containing the file pointed at by the current buffer. We then call our recr function. If it returns a directory we return that, if it returns nil however, we call the original erlang-flymake-get-app-dir function.
At this point we should have our project root. Now its a simple matter of recursively searching down the directory tree looking for files of a certain name. So we create a function that does just that, given a directory and a name will return a list of absolute paths for each subdirectory that matches the specified name.
(defun ebm-directory-dirs (dir name)
"Find all directories in DIR."
(unless (file-directory-p dir)
(error "Not a directory `%s'" dir))
(let ((dir (directory-file-name dir))
(dirs '())
(files (directory-files dir nil nil t)))
(dolist (file files)
(unless (member file '("." ".."))
(let ((absolute-path (expand-file-name (concat dir "/" file))))
(when (file-directory-p absolute-path)
(if (string= file name)
(setq dirs (append (cons absolute-path
(ebm-directory-dirs absolute-path name))
dirs))
(setq dirs (append
(ebm-directory-dirs absolute-path name)
dirs)))))))
dirs))
Now we write a couple of functions to replace the corresponding functions in `erlang-flymake`. The first looks for all `ebin` directories while the second looks for all `include` directories.
(defun ebm-get-deps-code-path-dirs ()
(ebm-directory-dirs (ebm-find-rebar-top) "ebin"))
(defun ebm-get-deps-include-dirs ()
(ebm-directory-dirs (ebm-find-rebar-top) "include"))
Finally we replace the `erlang-flymake` versions of those functions with our implementations.
(fset 'erlang-flymake-get-code-path-dirs 'ebm-get-deps-code-path-dirs) (fset 'erlang-flymake-get-include-dirs-function 'ebm-get-deps-include-dirs)
Conclusion
This approach is a bit of a hack, we basically use some heuristics to find a root and then just grab everything under that that looks remotely like a code or include directory. While its a bit hacky it has the valuable upside that its flexible and robust.
Common Test is a well thought out integration testing framework for Erlang. If you
are not using it you probably should be. However, it has one fault. It
does not return non-negative exit status’ to the caller when the tests
fail. This is a major oversight, and it makes it difficult to use as
part of a continuous integration scheme or in a make based build
system.
The long term fix is for the OTP folks to resolve the issue in the
ct_run command. To that end I have filed a bug report with the
Erlang folks. In the short term, though, we need this behaving
correctly. After much twiddling around with different solutions and
conversions on the erlang-questions list. This solution finally popped
out of a conversation with Lukas Larsson. Basically, we use the old
unix standby of awk.
ct_run -dir tests ... | awk "/FAILED/{exit 1;}/failed/{exit 1;}/SKIPPED/{exit 1;}"
Where ... is replaced with your additional options. Its not the best
solution on the planet, but it is the simplest one that I found that
works consistently.
