Deploy Erlang Target System to Heroku

In this post these new tools will be used:

First, clone minasan and create the Heroku application on Cedar-14:

$ git clone https://github.com/tsloughter/minasan.git  
$ cd minasan  
$ heroku create --stack cedar-14
Now that Heroku has the cedar-14 stack if you are also running a recent Linux distro you can upload the target system created by _relx_ directly to your app, before now we would have to build it on Heroku or in a system with an older glibc to work on Heroku's Ubuntu 10.04. Since _minasan_ is using binary packages and a fork of _rebar_ be sure to use the _rebar_ included in the repo, same goes for _relx _so that including the Procfile in the tarball works. The first step will be to update the package index for _rebar_, then compiling and building the release tarball (with _erts_ included and _dev-mode_ off so the Erlang runtime is included):
$ ./rebar update  
$ ./rebar compile  
$ ./relx -i true --dev-mode false release tar
Using the new Slug API endpoint through _hk slug_ the tarball can be pushed directly as a slug to your app and then scale up the web process to at least 1:
$ hk slug _rel/minasan/minasan-0.0.1.tar.gz  
$ hk scale web=1  
$ hk open  

Your browser should now open to your new app.

A few things to note:

  • './rebar pkgs' will show you a list of available packages to use in rebar deps
  • Currently 'hk slug' only support sending a tarball that does not yet have the structure of a slug. So it is unpacked and repacked. I plan to support directories and properly formatted tarballs.

Designing for Actor Based Systems

Many people are intrigued and excited about Erlang style concurrency. Once they get the capability in their hands though they realize that they don’t know how to take advantage of the capabilities processes or actors provide. To do this we need to understand how to decompose systems with process based concurrency in mind. Keep in mind, that this material works equally well for actors in Scala or agents in F#. Differences between actors and processes don't much matter for the sake of this discussion. Before we dive into process based design it will be helpful to look at a more familiar approach so we can contrast the two.

If you come from an OO background your natural instinct is to design much like you do when decomposing a problem for OO programming. After all, processes are much like objects in that they send messages to one another and they hold state. It goes something like this

  1. Determine your use cases
  2. Create a narrative of what it is you are trying to design
  3. Run through the narrative and pull out the nouns as potential classes
  4. Do the same for the verbs acting on the nouns as potential methods on the classes
  5. Clean all this up getting consolidating any duplication
    For example, lets say you were trying to build software to run a vending machine. The use cases might be paying for and getting a soda. Another one might be paying too little and getting change back. So one of the narratives there might be

As a customer I put sufficient coins into the vending machine and then press the selection button for coke and then press the button to vend and the robotic arm fetches a coke and dumps it into the pickup tray. The coke is nice and cold because the cooling system keeps the air in the vending machine at 50 degrees.

Now we think about all the unique nouns in our narrative which are: customer, coins, vending machine, selection button, coke, vend button, robotic arm and cooling system. And we generally turn them into objects. Next we consider the verbs that act on those nouns and consider them for methods.

selection button : push

vend button : push

robotic arm : pickup (coke)

etcetera... After this you apply some lovely object oriented design principles and voila - you have a system that nicely models your narrative but does not take advantage of more than a single core on your system and is positively undistributable.

Oh come on you say, don’t be daft, substitute the word object for actor and you are good to go. Well as it turns out not quite. Do we really need a coin process, how about a vend button process and lets not forget the coke process?? Darnit, this makes no sense! Lets back up and see what we can do about this.

Designing for Process Based Concurrency

The first thing you must do before we move on is say this three times

“Processes are not threads. Processes are really cheap. Ohmmmm”

“Processes are not threads. Processes are really cheap. Ohmmmm”

“Processes are not threads. Processes are really cheap. Ohmmmm”

This trips up folks new to process bases systems. They want to be stingy with processes worrying that they will take a long time to create, have massive context switching times, pollute L1 cache, etc… Remember that in almost all such systems, certainly for Erlang, Scala and F# processes/actors/agents are green threads. This means they have their own schedulers built into the VM they are running in. You never have to swap out a thread to switch between running one process vs another. With Erlang based systems you usually configure one erlang scheduler per core on the system. These schedulers remain relatively constant.

With that in mind we solve some of the sticking points many new to process based systems have; not taking advantage of all the concurrency in the system or using complex “process pooling”.

One process for each truly concurrent *activity* in the system.

That is the rule. Going back to our vending machine what do we have that is really concurrent in that system. Coins? Not really. Slots? Not really. Buttons? Not really. Those are not activities they are things. What are the truly concurrent activities, the activities that do not have to happen in synchronous lock step?

  • Putting coins into the slot
  • Handling coins
  • Handling selections
  • Fetching the coke and putting it into the pickup tray
  • Cooling the soda
    We can use a process for all of these activities. We don’t If you want to name them for the nouns that perform the activities - but remember we are not making them processes because they are nouns. Notice how we go granular here, we did not just create a process for the customer and the vending machine. We created one for all the truly concurrent activities in our narrative - in this way we leverage more of the concurrency available to us. Now we know what processes we need, the next step is to organize them.

Organizing Processes

The various languages that use process based concurrency have differing levels of sophistication here. I am going to draw on the concepts from the Erlang language which have been used in the Akka system for Scala and which I have rolled successfully myself in F#.

Again, forget all of your OO modeling techniques. Processes are not objects, they are fundamental units of concurrency. Forget all your thread modeling techniques - it’s not even close. Share nothing copy everything changes the game. To get started think about which of your processes have to cooperate with one another. In this case what do we have.

  • Putting coins in a slot, cooperates with
  • Handling coins, cooperates with
  • Handling selections, cooperates with
  • Fetching the coke and putting it into the pickup tray
    and nothing cooperates with, Cooling the soda - it happens whether or not other processes are there to support it or not. Putting coins in the slot however makes no sense if there is no way to handle them, and handing them makes no sense of you can’t make a selection and making a selection… well you get the drift.

To model this we are going to use a tree of “Supervisors”. Supervisors create and watch over processes. Because of copy everything share nothing properties of actors one can’t corrupt another. So, a supervisor can watch over an actor and restart it when it blows up in the presence of some error. This means we get some incredible fault tolerance. But, that aside, lets talk about how to model these dependencies. We do so in a tree. First, we setup a supervisor at the top of the tree which models no dependencies between any of the processes it starts. In this layer we add the cooling system and then we add another supervisor which will start the group of dependent processes in the order in which they depend on one another. This supervisor will restart processes that die according to their dependencies. If a dependency dies the supervisor will kill and restart dependent processes so that everything starts in a known base state down the chain.

**
proctree

**

Now with things decomposed into processes, dependencies fleshed out and placed into a supervision hierarchy you are basically ready to go. Is there more to design for actor based concurrency - yes of course there is but here you have the fundamentals. Now it’s time to go and play with it and generate questions. Feel free to ask them here or on twitter at @martinjlogan.  I may do a second installment on some more advanced topics based on feedback.

If you want to learn more come to Erlang Camp Oct 10 and 11, 2014 in Austin!

**

**

Erlang Postgres Connection Pool with Episcina

Almost exactly a year ago I was looking to merge the many forks of Will Glozer's Postgres client for use in a project at Heroku. Instead Semiocast released their client, I gave it a try and never looked back. (But note that David Welton, a braver person than me, is working on merging the forks of epgsql at this time). I found Semiocast's client to be clean, stable and I liked the interface better.

At the same time I was in the need of a connection pooler. Many have relied on poolboy or pooler for this purpose but neither actually fits the use case of connection pooling that well. Luckily Eric and Jordan were in the need at the same time and created the Erlware project episcina, which they based off of Joseph Wecker's fork of Will Glozer's epgsql pool. Episcina differs in that it is purely for connection pooling, it is not for pooling workers and it is not for pooling generic processes.

Here I'll show how I combined the two in a simple example.

To start we have a sys.config file to configure episcina:

{episcina, [{pools, [{primary,  
                        [{size, 10},                          
                         {timeout, 10000},  
                         {connect_provider, {pp_db, open,  
                                             [[{host, "localhost"}  
                                              ,{database, "postgres_pool"}  
                                              ,{port, 5432}  
                                              ,{user, "postgres"}  
                                              ,{password, "password"}]]}},  
                         {close_provider, {pp_db, close, []}}]}]  
              }  
             ]  
}  

A key thing to note here is the connect and close providers are function calls to modules within the project and not the Postgres client. Episcina requires a return value of {ok, pid()} and the Semiocast client returns {pgsql_connection, pid()}, so we wrap the connection calls to get around that:

-spec get_connection(atom()) -> {pgsql_connection, pid()} | {error, timeout}.  
get_connection(Pool) ->  
    case episcina:get_connection(Pool) of  
        {ok, Pid} ->  
            {pgsql_connection, Pid};  
        {error, timeout} ->  
            {error, timeout}  
    end.  

-spec return_connection(atom(), {pgsql_connection, pid()}) -> ok.  
return_connection(Pool, {pgsql_connection, Pid}) ->  
    episcina:return_connection(Pool, Pid).  

-spec open(list()) -> {ok, pid()}.  
open(DBArgs) ->  
    {pgsql_connection, Pid} = pgsql_connection:open(DBArgs),  
    {ok, Pid}.  

-spec close(pid()) -> ok.  
close(Pid) ->  
    pgsql_connection:close({pgsql_connection, Pid}).  

And here is the query function to get a connection and return it to the pool after completion:

  
-spec query(string()) -> tuple().  
query(Query) ->  
    C = get_connection(primary),  
    try  
        pgsql_connection:simple_query(Query, [], infinity, C)  
    after  
        return_connection(primary, C)  
    end.  

This example project uses relx to build a release which will start episcina on boot:

{release, {postgres_pool, "0.0.1"},  
  [postgres_pool]}.  

{sys_config, "./config/sys.config"}.  
{dev_mode, true}.  

{include_erts, true}.  
{extended_start_script, true}.  

Boot the release to an interactive shell and play around:

  
λ _rel/bin/postgres_pool console  
([email protected])1> pp_db:query("SELECT 1").  
{{select,1},[{1}]}  

Some Thoughts on Go and Erlang

UPDATE: I'm seeing that I did not make the point of this post clear. I am not saying Go is wrong or should change because it isn't like Erlang. What I am attempting to show is the choices Go made that make it not an alternative to Erlang for backends where availability and low latency for high numbers of concurrent requests is a requirement. And notice I'm not writing this about a language like Julia. I have heard Go pitched as an alternative to Erlang for not only new projects but replacing old. No one would say the same for Julia, but Go and Node.js are seen by some as friendlier alternatives. And no, Erlang isn't the solution for everything! But this is specifically about where Erlang is appropriate and Go is lacking.

I'm going to attempt to leave out my subjective opinions for disliking parts of Go, such as syntax or lack of pattern matching, and explain objective reasons for the language and runtime not being fit for certain types of systems. But I'll start with the good.

Where Go Shines

Clients

As Rob Pike wrote, his biggest surprise was Go is mostly gaining developers from Python and Ruby, not C++. To me this trend has been great to see. No more slow clients installed through pip or gems! (Though for some reason Node.js for clients is growing, wtf Keybase?)

Go provides developers with a fast and easy to use high level, statically typed language with garbage collection and concurrency primitives. It would be great for C++ developers to move to Go as well, the programs that crash constantly on my machine are proprietary C++ that love to misuse memory -- Hipchat and Spotify. But Rob Pike pointed out that the C++ developers don't want the simplified, yet powerful, world of Go. Ruby and Python developers, rightly, do.

Tooling

Getting up and going with building executables depending on third party libraries can be done easily without depending on third party tools, it all comes with Go. While the tools aren't perfect, there are tools to fill in some gaps like Godep, it is still a huge win for the language.

Where Go Falls Short

Some of Go's design decisions are detrimental when it comes to writing low-latency fault-tolerant systems.

Concurrency

Yes, I listed concurrency primitives as a plus in the first section. It is in the case of replacing Ruby or Python or C++ for clients. But when it comes to complex backends that need to be fault-tolerant Go is as broken as any other language with shared state.

Pre-emptive Scheduling

Here Go has gotten much better. Go's pre-emptive scheduling was done on syscalls but now pre-emption is done when a goroutine checks the stack, which it does on every function call, and this may be marked to fail (causing pre-emption) if the goroutine has been running for longer than some time period. While this is an improvement it still lags behind Erlang's reduction counting and newly added dirty schedulers for improved integration with C.

Garbage Collection

In Go garbage collection is global mark and sweep. This pauses all goroutines during the sweep and this is terrible for latency. Again, low latency is hard, the more the runtime can do for you the better.

Error Handling

This isn't just about having no exceptions and the use of checking if a second return value is nil. Goroutines have no identity which means Go lacks the ability to link or monitor goroutines. No linking (instead using panic and defer) and no process isolation means you can not fall back on crashing and restarting in a stable state. There will be bugs in production and a lot of those bugs will be Heisenbugs, so being able to layout processes, isolated from each other but linked based on their dependencies, is key for fault tolerance.

And on top of these major omissions in dealing with faults Go has nil. How in 2014 this is considered OK I haven't wrapped my mind around yet. I'll just leave it at that, with a befuddled look.

Introspection

Not having a REPL is annoying for development, but no remote shell for running systems is a deal breaker. Erlang has impressive tracing capabilities and tools built on those capabilities like recon_trace. Erlang's introspection greatly improves development as well as maintenance of complex running systems.

Static Linking

Yes, another thing that was also in the positives but becomes a negative when used in systems that are expected to be long running. While having no linking does means slower execution it gives Erlang advantages in the case of code replacement on running systems. It is important to note that due to Erlang's scheduling and garbage collecting strategies many of these speed tradeoffs do not mean Erlang will be slower than an implementation in another language, especially if the Erlang implementation is the only one still running.

Code Organization

The OTP framework provides libraries for common patterns. OTP not only means less code to write and better abstractions but also improves readability. Following the OTP standards with applications, supervisors and workers (genserver, genfsm, gen_event) means a developer new to the program is able to work down through the tree of processes and how they interact. Go's channels, unidentifiable goroutines and lack of patterns to separate goroutines into separate modules leads to much harder code to reason about.

Can or Even Should Go Change?

Erlang has been around for decades and Go is the new kid on the block, so can Go improve in these areas? Some, yes, but most of it can not because of the choices made in the design of the language which were not fault tolerance and low latency.

This doesn't mean Go is "bad" or "wrong". It simply makes different choices and thus is better suited for different problems than a language like Erlang.

How to use Vim for Erlang Development

vim editor logo

This post sponsored by ErlangCamp 2013 in Nashville which was epic!

You are about to learn to use Vim as your editor for Erlang development. You will learn how to install and use a variety of really powerful Vim plugins to make Erlang dev with Vim smooth and satisfying!

I have been developing Erlang now for about 13 years, many of them full time and even wrote a book on Erlang: Erlang & OTP in Action. I have loved every minute of it but there was always one thing that made me sad, probably makes you sad too - Emacs. Emacs is the de-facto editor for Erlang. The emacs mode included with the Erlang distro is quite wonderful. The fact still remains, Emacs, we do not like it. ctrl ~, ctrl x ctrl f etc... Nope!

Setting up Vim for Erlang

Let's get started setting up Vim for Erlang development. The first thing we need to do is setup pathogen so that installing subsequent packages is really simple. The first thing to do is create the directory $HOME/.vim/autoload. Download pathogen.vim from here and place it into this directory. Now add the following 2 commands to your $HOME/.vimrc file.

call pathogen#infect()
call pathogen#helptags()

At this point pathogen will install and generate help documentation for any plugin you place into the $HOME/.vim/bundle directory - which you should of course create.

With this created now we are ready to start installing plugins to make your life easier. Try these on for size by cloning these git repos directly into the $HOME/.vim/bundle directory. They will simply work next time you start vim.

[**vimerl.vim**](https://github.com/jimenezrick/vimerl)Indenting, autocomplete and more for Erlang
[**ctrlp.vim**](https://github.com/kien/ctrlp.vim)ctrl p and open a powerful fuzzy file finder. Makes navigating file trees a thing of the past.
[**NERDTree**](https://github.com/scrooloose/nerdtree)Powerful file tree navigator right in vim - don't use it much since I installed ctrlp though.
[**NERDTree Tabs**](https://github.com/jistr/vim-nerdtree-tabs)Add the NERDTree file finder to all tabs you have open in vim.

Before we get into basics on how to use all these plugins to create Erlang magic I want to show you two bonus tricks I really love. First, get a better color scheme. To do this create the directory $HOME/.vim/colors and find yourself a slick color scheme to drop into it. I recommend vividchalk.vim by TPope.

Pro Tip

For dropbox or other file sync users keep all your vim installs in sync easily like so; take your .vim and your .vimrc and move them into your Dropbox directory. Then run:

`

ln -s ~/Dropbox/.vim ~/.vim

ln -s ~/Dropbox/.vimrc ~/.vimrc

`

Now all your machines vim installs will run just the same. If you have compatibility problems on any one, well then just skip this for that machine.

Ok, so now on to how to use these plugins for Erlang/Vim greatness.

How to Use our Vim Plugins for Erlang Dev

I am going to use the source for Erlware Commons as an example. So I clone it first and then change into the erlware_commons directory and run vim. Now lets say I know what file I want to update, specifically the "ecdate.erl" file. The first thing I do is type p and then start typing ecdate.erl.

                                                                                                                                                            
~                                                                                 
[No Name] [TYPE= unix] [0/1 (100%)]                                               
> test/ec_dictionary_proper.erl  
> src/ec_dictionary.erl  
> src/ec_date.erl                                                                 
 prt  path  ={ files }=  >> ec_da  

You can see that as I start typing and get to "ecda" ctrlp has already displayed a narrowed down list of files in the directory tree under where I have opened vim that match. The file on the bottom ecdate.erl is the one selected and so just pressing enter here will open it. If I wanted to select "test/ecdictionaryproper.erl" then I could simply press the up arrow and select it or keep typing until it was the only selection.

Now, what if I don't know what file I want to select? This is where NERDTree comes into play. Run :NERDTree and you will pop open the file browser. Like this:

  
  Press ? for help             |  
                               |~                                                 
.. (up a dir)                  |~                                                 
<lang-projects/erlware_commons/|~                                                 
▸ doc/                         |~                                                 
▸ priv/                        |~                                                 
▸ src/                         |~                                                 
▸ test/                        |~                                                 
  CONTRIBUTING.md              |~                                                 
  COPYING                      |~                                                 
  Makefile                     |~                                                 
  README.md                    |~                                                 
  rebar.config                 |~                                                 
  rebar.config.script          |~                                                 
~                              |~                                                 
~                              |~                                                 
~                              |~                                                                                                                                         

Here we can see the directory tree for Erlware Commons. Each of the directories can be easily selected and expanded. Individual files can be selected and opened. There are a variety of ways to open a file. Below are the most common:

  • &lt;enter&gt; will open the file in the right pane
  • T will open in a new tab within vim and keep focus in NERDTree
  • t will open in a new tab and bring focus to the new tab

IF you want to see the NERDTree browser in all your tabs use :NERDTreeTabsToggle to toggle it on and off. It will be the exact same NERDTree in the exact same state and cursor position on all tabs - nice! Once you are focused on the code in a given tab and you want to jump back to the left and into the NERDTree pane use &lt;ctrl&gt; ww

Once you have a load of tabs open you need to switch between then and to do this you need only two commands:

  • gt will goto the next tab the next tab
  • gT will goto the previous tab

Pro Tip

Map the tab commands and the NERDTreeTabsToggle command by adding the following to your vimrc.

`

map <C-t> :tabn<Enter>

map <C-n> :tabnew<Enter>

map nt :NERDTreeTabsToggle<Enter>

`

Ok, now on to editing Erlang with vimerl.

Editing with vimerl

This is not going to be an exhaustive list of vimerl editing commands but just a few of the goodies. The 20% you will use 80% of the time.

Auto-indenting

vimerl will auto-indent for you as you type. But if you come across a line that you want to indent try typing ==. Lets say you want to indent a block of code. Simple, mark the line that starts the block with ma then go to the end of the block and tell vimerl to indent to the mark as such: ='a. Now if your whole file is a mess then try gg to go to the beginning of your file and then =G to indent all the way to the end. You can do this all in one step as in gg=G.

Code Completion

ctrl-x ctrl-o after typing a module name and a : will cause vimerl to suggest function names for you. It does this by searching the .beam and .erl files in the erlang code path (code:get_path() to see what they are) as well as looking at your rebar deps_dir if you are using rebar.config as part of your project.

Skeletons

This is the feature that I loved most about the emacs mode for Erlang, well this and the auto indenting (most of the time, the fun() indenting still feels like a kick in the teeth). Here is a list of the most useful skeletons and the commands to generate them from within vimerl.

  • :ErlangApplication generate the skeleton for an OTP application behaviour.
  • :ErlangSupervisor generate the skeleton for an OTP supervisor behaviour.
  • :ErlangGen[Server|Fsm|Event] skeletons for gen server, fsm and event - yay!

Brilliant isn't it. Before I let you go there is one more invaluable command you should know about which is :help vimerl which will give you a list of all the other useful commands you may want to use. Remember to get it working be sure to add call pathogen#helptags() to the top of your .vimrc file. Goodbye Emacs, welcome back old friend Vim.

Follow me on twitter @martinjlogan

<esc>:wq