Erlang

Erlang Postgres Connection Pool with Episcina

Almost exactly a year ago I was looking to merge the many forks of Will Glozer's Postgres client for use in a project at Heroku. Instead Semiocast released their client, I gave it a try and never looked back. (But note that David Welton, a braver person than me, is working on merging the forks of epgsql at this time). I found Semiocast's client to be clean, stable and I liked the interface better.

At the same time I was in the need of a connection pooler. Many have relied on poolboy or pooler for this purpose but neither actually fits the use case of connection pooling that well. Luckily Eric and Jordan were in the need at the same time and created the Erlware project episcina, which they based off of Joseph Wecker's fork of Will Glozer's epgsql pool. Episcina differs in that it is purely for connection pooling, it is not for pooling workers and it is not for pooling generic processes.

Here I'll show how I combined the two in a simple example.

To start we have a sys.config file to configure episcina:

{episcina, [{pools, [{primary,  
                        [{size, 10},                          
                         {timeout, 10000},  
                         {connect_provider, {pp_db, open,  
                                             [[{host, "localhost"}  
                                              ,{database, "postgres_pool"}  
                                              ,{port, 5432}  
                                              ,{user, "postgres"}  
                                              ,{password, "password"}]]}},  
                         {close_provider, {pp_db, close, []}}]}]  
              }  
             ]  
}  

A key thing to note here is the connect and close providers are function calls to modules within the project and not the Postgres client. Episcina requires a return value of {ok, pid()} and the Semiocast client returns {pgsql_connection, pid()}, so we wrap the connection calls to get around that:

-spec get_connection(atom()) -> {pgsql_connection, pid()} | {error, timeout}.  
get_connection(Pool) ->  
    case episcina:get_connection(Pool) of  
        {ok, Pid} ->  
            {pgsql_connection, Pid};  
        {error, timeout} ->  
            {error, timeout}  
    end.  

-spec return_connection(atom(), {pgsql_connection, pid()}) -> ok.  
return_connection(Pool, {pgsql_connection, Pid}) ->  
    episcina:return_connection(Pool, Pid).  

-spec open(list()) -> {ok, pid()}.  
open(DBArgs) ->  
    {pgsql_connection, Pid} = pgsql_connection:open(DBArgs),  
    {ok, Pid}.  

-spec close(pid()) -> ok.  
close(Pid) ->  
    pgsql_connection:close({pgsql_connection, Pid}).  

And here is the query function to get a connection and return it to the pool after completion:

  
-spec query(string()) -> tuple().  
query(Query) ->  
    C = get_connection(primary),  
    try  
        pgsql_connection:simple_query(Query, [], infinity, C)  
    after  
        return_connection(primary, C)  
    end.  

This example project uses relx to build a release which will start episcina on boot:

{release, {postgres_pool, "0.0.1"},  
  [postgres_pool]}.  

{sys_config, "./config/sys.config"}.  
{dev_mode, true}.  

{include_erts, true}.  
{extended_start_script, true}.  

Boot the release to an interactive shell and play around:

  
λ _rel/bin/postgres_pool console  
(postgres_pool@127.0.0.1)1> pp_db:query("SELECT 1").  
{{select,1},[{1}]}  

Some Thoughts on Go and Erlang

UPDATE: I'm seeing that I did not make the point of this post clear. I am not saying Go is wrong or should change because it isn't like Erlang. What I am attempting to show is the choices Go made that make it not an alternative to Erlang for backends where availability and low latency for high numbers of concurrent requests is a requirement. And notice I'm not writing this about a language like Julia. I have heard Go pitched as an alternative to Erlang for not only new projects but replacing old. No one would say the same for Julia, but Go and Node.js are seen by some as friendlier alternatives. And no, Erlang isn't the solution for everything! But this is specifically about where Erlang is appropriate and Go is lacking.

I'm going to attempt to leave out my subjective opinions for disliking parts of Go, such as syntax or lack of pattern matching, and explain objective reasons for the language and runtime not being fit for certain types of systems. But I'll start with the good.

Where Go Shines

Clients

As Rob Pike wrote, his biggest surprise was Go is mostly gaining developers from Python and Ruby, not C++. To me this trend has been great to see. No more slow clients installed through pip or gems! (Though for some reason Node.js for clients is growing, wtf Keybase?)

Go provides developers with a fast and easy to use high level, statically typed language with garbage collection and concurrency primitives. It would be great for C++ developers to move to Go as well, the programs that crash constantly on my machine are proprietary C++ that love to misuse memory -- Hipchat and Spotify. But Rob Pike pointed out that the C++ developers don't want the simplified, yet powerful, world of Go. Ruby and Python developers, rightly, do.

Tooling

Getting up and going with building executables depending on third party libraries can be done easily without depending on third party tools, it all comes with Go. While the tools aren't perfect, there are tools to fill in some gaps like Godep, it is still a huge win for the language.

Where Go Falls Short

Some of Go's design decisions are detrimental when it comes to writing low-latency fault-tolerant systems.

Concurrency

Yes, I listed concurrency primitives as a plus in the first section. It is in the case of replacing Ruby or Python or C++ for clients. But when it comes to complex backends that need to be fault-tolerant Go is as broken as any other language with shared state.

Pre-emptive Scheduling

Here Go has gotten much better. Go's pre-emptive scheduling was done on syscalls but now pre-emption is done when a goroutine checks the stack, which it does on every function call, and this may be marked to fail (causing pre-emption) if the goroutine has been running for longer than some time period. While this is an improvement it still lags behind Erlang's reduction counting and newly added dirty schedulers for improved integration with C.

Garbage Collection

In Go garbage collection is global mark and sweep. This pauses all goroutines during the sweep and this is terrible for latency. Again, low latency is hard, the more the runtime can do for you the better.

Error Handling

This isn't just about having no exceptions and the use of checking if a second return value is nil. Goroutines have no identity which means Go lacks the ability to link or monitor goroutines. No linking (instead using panic and defer) and no process isolation means you can not fall back on crashing and restarting in a stable state. There will be bugs in production and a lot of those bugs will be Heisenbugs, so being able to layout processes, isolated from each other but linked based on their dependencies, is key for fault tolerance.

And on top of these major omissions in dealing with faults Go has nil. How in 2014 this is considered OK I haven't wrapped my mind around yet. I'll just leave it at that, with a befuddled look.

Introspection

Not having a REPL is annoying for development, but no remote shell for running systems is a deal breaker. Erlang has impressive tracing capabilities and tools built on those capabilities like recon_trace. Erlang's introspection greatly improves development as well as maintenance of complex running systems.

Static Linking

Yes, another thing that was also in the positives but becomes a negative when used in systems that are expected to be long running. While having no linking does means slower execution it gives Erlang advantages in the case of code replacement on running systems. It is important to note that due to Erlang's scheduling and garbage collecting strategies many of these speed tradeoffs do not mean Erlang will be slower than an implementation in another language, especially if the Erlang implementation is the only one still running.

Code Organization

The OTP framework provides libraries for common patterns. OTP not only means less code to write and better abstractions but also improves readability. Following the OTP standards with applications, supervisors and workers (genserver, genfsm, gen_event) means a developer new to the program is able to work down through the tree of processes and how they interact. Go's channels, unidentifiable goroutines and lack of patterns to separate goroutines into separate modules leads to much harder code to reason about.

Can or Even Should Go Change?

Erlang has been around for decades and Go is the new kid on the block, so can Go improve in these areas? Some, yes, but most of it can not because of the choices made in the design of the language which were not fault tolerance and low latency.

This doesn't mean Go is "bad" or "wrong". It simply makes different choices and thus is better suited for different problems than a language like Erlang.

Projmake-mode: Flymake Replacement

There is a great new Emacs plugin from Eric Merritt that like FlyMake builds your code and highlights within Emacs any errors or warnings, but unlike FlyMake builds across the whole project. You can clone the mode from here projmake-mode

After cloning the repo to your desired location add this bit to your dot emacs file, replacing <PATH> with the path to where you cloned the repo.

[gist]3794732[/gist]

This Emacs code also uses add-hook to set projmake-mode to start for erlang-mode is loaded. Projmake by default knows how to handle rebar and Make based builds so there is no setup after this, assuming your project is built this way.

Here is my Makefile for building Erlang projects with rebar, replace PROJECT with the name of your project:

[gist]3795007[/gist]

Now you can load Emacs and a file from your project and if it is an Erlang file due to the add-hook function in our dot emacs file it will automatically load projmake-mode. You can add hooks for other modes or simply run M-x projmake-mode.

For more documentation and how to extend to other types of projects check out the documentation.

Maru Models: JSON to Erlang Record with Custom Types

Working with Erlang for writing RESTful interfaces JSON is the communication "language" of choice. For simplifying the process of JSON to a model the backend could work with efficiently I've created marumodels. This app decodes the JSON with _jiffy and uses functions generated by a modified version of Ulf's exprecs to create an Erlang record. The generated functions are created with type information from the record definition and when a property is set for the record through these functions it is first passed to the convert function of marumodeltypes to do any necessary processing.

I separated this application into a separate repo to simplify people trying the examples. But the real development will be done in the Maru main repo.

Getting Flymake and Rebar to Play Nice

TLDR;
Copy and paste the following into your elisp erlang-mode configuration to get flymake working with Rebar projects.

  
 (defun ebm-find-rebar-top-recr (dirname)  
      (let* ((project-dir (locate-dominating-file dirname "rebar.config")))  
        (if project-dir  
            (let* ((parent-dir (file-name-directory (directory-file-name project-dir)))  
                   (top-project-dir (if (and parent-dir (not (string= parent-dir "/")))  
                                       (ebm-find-rebar-top-recr parent-dir)  
                                      nil)))  
              (if top-project-dir  
                  top-project-dir  
                project-dir))  
              project-dir)))  

    (defun ebm-find-rebar-top ()  
      (interactive)  
      (let* ((dirname (file-name-directory (buffer-file-name)))  
             (project-dir (ebm-find-rebar-top-recr dirname)))  
        (if project-dir  
            project-dir  
          (erlang-flymake-get-app-dir))))  

     (defun ebm-directory-dirs (dir name)  
        "Find all directories in DIR."  
        (unless (file-directory-p dir)  
          (error "Not a directory `%s'" dir))  
        (let ((dir (directory-file-name dir))  
              (dirs '())  
              (files (directory-files dir nil nil t)))  
            (dolist (file files)  
              (unless (member file '("." ".."))  
                (let ((absolute-path (expand-file-name (concat dir "/" file))))  
                  (when (file-directory-p absolute-path)  
                    (if (string= file name)  
                        (setq dirs (append (cons absolute-path  
                                                 (ebm-directory-dirs absolute-path name))  
                                           dirs))  
                        (setq dirs (append  
                                    (ebm-directory-dirs absolute-path name)  
                                    dirs)))))))  
              dirs))  

    (defun ebm-get-deps-code-path-dirs ()  
        (ebm-directory-dirs (ebm-find-rebar-top) "ebin"))  

    (defun ebm-get-deps-include-dirs ()  
       (ebm-directory-dirs (ebm-find-rebar-top) "include"))  

    (fset 'erlang-flymake-get-code-path-dirs 'ebm-get-deps-code-path-dirs)  
    (fset 'erlang-flymake-get-include-dirs-function 'ebm-get-deps-include-dirs)  

Intro

Its probably no great surprise to anyone that I dislike Rebar a lot. That said there are times when I have no choice but to use it. This is always either because a company I am contracting for uses it, or an open source project I am contributing to uses it. When I am forced to use it there are a few things I don't want to give up. Most important among these is Flymake for Erlang. The default setup for Flymake doesn't work for Rebar projects because Flymake does not know where the code and include paths for dependencies are. Fortunately, we can fix this with a few lines of elisp.

Flymake For Erlang

First make sure you have Flymake for Erlang installed. It is easiest just to follow the directions available on the Erlang Website.

The Elisp Additions for Erlang Flymake

There are two defvars that point to functions that are used to search for the correct code paths and include paths respectively. We are going to replace those functions with our own functions. Both these functions search upwards from the directory that contains the file pointed to by the current buffer, looking for the top most 'rebar.config' in the directory path. It then uses that for a base and searches down the directory structure looking for either 'ebin' files or 'include' files.

There are two things to note here. The first is that you must have already run get-deps for rebar for this to work and the second is that if your project is truly huge or you have way more dependencies then you probably need this search could take a second or two. That is a second or two too long in an interactive compiler like Flymake. That said, the likelihood that you will run into this second problem is quite low.

Getting Started

The very thing you want to do is ensure that you have required the erlang-flymake module. Most of what we do below depends on this.

  
(require 'erlang-flymake)  

Finding the Top rebar.config

The second thing we want to do is look for the top rebar.config in the project. If a rebar project contains more then one OTP application its quite likely that it will contain more then one rebar.config. The very topmost rebarconfig` is the right one to serve as root of our search. So we introduce a set of recursive functions to look for that top level dir.

  
    (defun ebm-find-rebar-top-recr (dirname)  
      (let* ((project-dir (locate-dominating-file dirname "rebar.config")))  
        (if project-dir  
            (let* ((parent-dir (file-name-directory (directory-file-name project-dir)))  
                   (top-project-dir (if (and parent-dir (not (string= parent-dir "/")))  
                                       (ebm-find-rebar-top-recr parent-dir)  
                                      nil)))  
              (if top-project-dir  
                  top-project-dir  
                project-dir))  
              project-dir)))  

ebm-find-rebar-top-recr will return either the top most directory or nil. Our next function takes that result and does something useful with.

  
    (defun ebm-find-rebar-top ()  
      (interactive)  
      (let* ((dirname (file-name-directory (buffer-file-name)))  
             (project-dir (ebm-find-rebar-top-recr dirname)))  
        (if project-dir  
            project-dir  
          (erlang-flymake-get-app-dir))))  

In this function, we get the directory containing the file pointed at by the current buffer. We then call our recr function. If it returns a directory we return that, if it returns nil however, we call the original erlang-flymake-get-app-dir function.

At this point we should have our project root. Now its a simple matter of recursively searching down the directory tree looking for files of a certain name. So we create a function that does just that, given a directory and a name will return a list of absolute paths for each subdirectory that matches the specified name.

  
(defun ebm-directory-dirs (dir name)  
        "Find all directories in DIR."  
        (unless (file-directory-p dir)  
          (error "Not a directory `%s'" dir))  
        (let ((dir (directory-file-name dir))  
              (dirs '())  
              (files (directory-files dir nil nil t)))  
            (dolist (file files)  
              (unless (member file '("." ".."))  
                (let ((absolute-path (expand-file-name (concat dir "/" file))))  
                  (when (file-directory-p absolute-path)  
                    (if (string= file name)  
                        (setq dirs (append (cons absolute-path  
                                                 (ebm-directory-dirs absolute-path name))  
                                           dirs))  
                        (setq dirs (append  
                                    (ebm-directory-dirs absolute-path name)  
                                    dirs)))))))  
              dirs))  

Now we write a couple of functions to replace the corresponding functions in erlang-flymake. The first looks for all ebin directories while the second looks for all include directories.

  
    (defun ebm-get-deps-code-path-dirs ()  
        (ebm-directory-dirs (ebm-find-rebar-top) "ebin"))  

    (defun ebm-get-deps-include-dirs ()  
       (ebm-directory-dirs (ebm-find-rebar-top) "include"))  

Finally we replace the erlang-flymake versions of those functions with our implementations.

  
(fset 'erlang-flymake-get-code-path-dirs 'ebm-get-deps-code-path-dirs)  
(fset 'erlang-flymake-get-include-dirs-function 'ebm-get-deps-include-dirs)  

Conclusion

This approach is a bit of a hack, we basically use some heuristics to find a root and then just grab everything under that that looks remotely like a code or include directory. While its a bit hacky it has the valuable upside that its flexible and robust.