Opa

Running Opa Applications on Heroku

TL;DR

As I've mentioned before, Opa is a new web framework that introduces not only the framework itself but a whole new language. A lot has changed in Opa since I last posted about it. Now Opa has a Javascript-esque look and runs on Node.js. But it still has the amazing typing system that makes Opa a joy to code in.

The currently available Heroku buildpack for Opa only supported the old, pre-Node, support. So I've created an all new buildpack and here I will show both a bit of how I created that buildpack and how to use it to run your Opa apps on Heroku.

The first step was creating a tarball of Opa that would work on Heroku. For this I used the build tool vulcan. Vulcan is able to build software on Heroku in order to assure what is built will work on Heroku through your buildpacks.

vulcan build -v -s ./opalang/ -c "mkdir /app/mlstate-opa && yes '' | ./opa-1.0.7.x64.run" -p /app/mlstate-opa

This command is telling vulcan to build what is in the directory opalang with a command that creates the directory /app/mlstate-opa and then runs the Opa provided install script to unpack the system. This is much simpler than building Opa from source, but it is still necessary to still use vulcan to create the tarball from the output of the install script to ensure paths are correct in the Opa generated scripts.

After this run, by vulcan's default, we will have /tmp/opalang.tgz. I upload this to S3, so that our buildpack is able to retrieve it.

Since Opa now relies on Node.js, the new buildpack must install both Node.js and the opalang.tgz that was generated. To do this I simply copied from the Node.js buildpack.

If you look at the Opa buildpack you'll see, as with any buildpack, it consists of three main scripts under ./bin/: compile, detect and release. There are three important parts for understanding how your Opa app must be changed to be supported by the buildpack.

First, the detect script relies on there being a opa.conf to detect this being an Opa application. This for now is less important since we will be specifying the buildpack to use to the heroku script. Second, in the compile script we rely on there being a Makefile in your application for building. There is no support for simply running opa to compile the code in your tree at this time. Third, since Opa relies on Node.js and Node modules from npm you must provide a package.json file that the compile script uses to install the necessary modules.

To demostrate this I converted Opa's hello_chat example to work on Heroku, see it on Github here.

There are two necessary changes. One, add the Procfile. A Procfile define the processes required for your application and how to run them. For hellochat_ we have:

web: ./hello_chat.exe --http-port $PORT

This tell Heroku that our web process is run from the binary hellochat.exe_. We must pass the _$PORT _variable to the Opa binary so that it binds to the proper port that Heroku expects it to be listening on to route our traffic.

Lastly, a package.json file is added so that our buildpack's compile script installs the necessary Node.js modules:

{  
  "name": "hello_chat",  
  "version": "0.0.1",  
  "dependencies": {  
      "mongodb" : "*",  
      "formidable" : "*",  
      "nodemailer" : "*",  
      "simplesmtp" : "*",  
      "imap" : "*"  
  },  
  "engines": {  
    "node": "0.8.7",  
    "npm": "1.1.x"  
  }  
}

With these additions to hellochat_ we are ready to create an Opa app on Heroku and push the code!

$ heroku create --stack cedar --buildpack https://github.com/tsloughter/heroku-buildpack-opa.git  
$ git push heroku master

The output from the push will show Node.js and npm being install, followed by Opa being unpacked and finally make being run against hellochat. The web process in _Procfile will then be run and the output will provide a link to go to our new application. I have the example running at http://mighty-garden-9304.herokuapp.com

Next time I'll delve into database and other addon support in Heroku with Opa applications.

Opa Plugin Development with a PubNub Example

This will be a two part series of posts on writing plugins for Opa. One of Opa's greatest features is you write everything in the functional, staticly typed Opa language. This even includes the frontend code you'd usually do in Javascript. This means your code is less error-prone which significantly reduces the amount of time spent on debugging code.

But how do you use Javascript library X within your Opa code?

While you can include Javascript code in a resource, like we do here for Google Analytics:

do Resource.register_external_js("/resources/js/google_analytics.js")

This isn't what we want to do if we want to call external Javascript functions from Opa code. For this we create an Opa plugin. In this post I'll be showing how I built the Opa client side bindings for the PubNub Javascript API, fork it on github and see all the other language bindings. PubNub is a cloud hosted real-time publish and subscribe interface for all types of applications.

Opa allows binding to external functions through its Binding System Library (BSL). In order to use this to create a plugin for some external Javascript, we create a file pubnub.js that we will compile to a plugin using opa-plugin-builder. Below is an example of one of the external functions that needs to be defined to use PubNub's publish.

##register publish : string, string -> void  
##args(c, m)  
{  
    PUBNUB.publish({  
                       channel : c, message : m  
                   });  

    return js_void;  
}

The first line registers the function publish and specifies its arguments to be two strings and its return value to be void. The next line names the arguments. You can think of ##args like Javascript's function keyword. In the function, we can use the PUBNUB library just like we would normally. Here we pass to publish the channel and the message to be published. Note that the message can be any Javascript object, even though its a string to the plugin. We simply must serialize the JSON before passing to publish, which we'll see shortly.

Now after defining external functions for all PubNub api calls in the pubnub.js file we can compile it:

$ opa-plugin-builder src/plugin/pubnub.js -o pubnub

This creates the plugin pubnub.opp.

To make using the plugin easier, we create a package in pubnub.opa that will contain a module PubNub. Below is the contents of pubnub.opa for the publish function that will show how to register the external functions and create a module named PubNub around them.

package pubnub  

pubnub_publish = %%pubnub.publish%%  

@client publish_client = pubnub_publish  

PubNub = {{  
  publish(channel: string, j : RPC.Json.json) =  
    publish_client(channel, Json.to_string(j))  
}}  

do Resource.register_external_js("http://cdn.pubnub.com/pubnub-3.1.min.js")

The first thing to notice is setting the pubnubpublish_ variable. The %% syntax is specifying that pubnub.publish is the key of an external primitive. pubnub is the name of the plugin (pubnub.opp) and publish is the function we registered in the plugin.

After this we create another variable publishclient_ that specifies that that this function (variables can contain functions) is to be run on the client side. Lastly, we define a module PubNub and a publish function inside the module to be used by our Opa programs.

At the end we make sure that the needed PubNub Javascript file is includes in all resources by registering it and giving the URL to the PubNub CDN.

Now we can compile it all together with the opa-plugin-builder command from above and use the plugin.

import pubnub  

broadcast(author, msg) =  
   username : RPC.Json.json = {String = author}  
   message : RPC.Json.json = {String = msg}  
   record_json : RPC.Json.json = {Record = [("username", username), ("message", message)]}  
   do PubNub.publish("chat", record_json)  
   Dom.clear_value(#entry)

The broadcast function is part of pubnubchat.opa. _broadcast takes 2 arguments, the username of the user sending the message and the message itself. We then construct three variables of Opa's json type, username, message and recordjson. _username and message are simply Json strings and they are then combined into a Json Record type that is a list of string/Json pairs. Since our PubNub.publish function converts the Opa Json to a string before sending to the Javascript binding we can simply pass the Opa Json variable recordjson_.

Lastly, lets take a quick look at the subscribe function. Defining it in pubnub.js look like:

##register subscribe : string, (string -> void) -> void

Since it takes a callback function you'll notice that the second argument's type looks different from in publish. (string -> void) is defining that the function takes a variable that is a function which takes a single argument of type string and has a void return type.

Now in pubnub.opa we have a subscribe function that is hardly different from the publish function we described above.

subscribe(channel: string, callback) =  
  subscribe_client(channel, callback)

But how do we define a function we can pass to subscribe? That is the great thing, it can be any Opa function!

PubNub.subscribe("chat", (x -> user_update(x)))

Above we are subscribing to the channel chat and passing an anonymous function that takes a single argument and sends it to the function userupdate_.

user_update(x) =  
  match Json.deserialize(x) with  
         | {some={Record = [("username", username), ("message", message)]}} ->  
                    line = <div class="line"> <div class="user">{Json.to_string(username)}:</>  
                             <div class="message">{Json.to_string(message)}</>  
                            </>  
                    do Dom.transform([#conversation +<- line ])  
                    Dom.scroll_to_bottom(#conversation)  
         | _ -> void

Now every time a message is posted to the chat channel, our callback, the anonymous function, will be run and call the function userupdate_ which we see deserializes the Json string to get an Opa Json Record. The rest is basic Opa DOM manipulation to add the new message to the messages in the conversation element.

In my next post on Opa and PubNub, I'll be describing an Opa server side API for PubNub.

Major OpaDo Speed-Up with @publish

I received a patch for OpaDo from Francois Regis Sinot that has made the hosted OpaDo http://opado.org on Dotcloud MUCH faster, adding and removing item is now instantaneous. He placed @publish around explicitly server side functions that caused adding todo items to go from 21 to 1 server calls. I thought this simple but maybe not obvious for new Opa developers (it certainly wasn't to me!) needed a blog post so that beginners like my self may find it and be able to start using the @publish directive. Check out the Opa manual for some more information on @publish, specifically I'd look at Section 4.7.4 and 10.2

Below are the changes he made. Which you can also of see in the commit diff on Github and check out the entire source.

 
- @async
- dbmakedone(username: string, id: string) =
+ @publish @async
+ dbmakedone(id: string) =
+ username = User.getusername()
items = /todo
items[username]
item = Option.get(StringMap.get(id, items))
/todo_items[username] <- StringMap.add(id, {item with done=true}, items)

- @async
+ @publish @async
dbremoveitem(id: string) =
username = User.getusername()
items = /todo
items[username]

- @async
+ @publish @async
dbaddtodo(id: string, x: string) =
username = User.getusername()
items = /todo
items[username]
/todoitems[username] <- StringMap.add(id, { value=x done=false createdat="" }, items)

+ @publish
addtodos() =
username = User.get
username()
items = /todo_items[username]

Adding JS To all Opa Resources: Use Case Google Analytics

I decided I wanted to add Google Analytics to OpaDo but had no idea how to easily tell each page to include the necessary Javascript. I asked on the Opa mailing list and got a quick and simple response. Frederic Ye pointed me to Resource.registerexternaljs

It couldn't have been any easier. You simply place your googleanalytics.js file in your project and use the Resource.registerexternal_js function to modify the default customization of all Resources. See code below or on the github repo.

package opado.main  

import opado.user  
import opado.admin  
import opado.todo  

urls : Parser.general_parser(http_request -> resource) =  
  parser  
  | {Rule.debug_parse_string(s -> Log.notice("URL",s))} Rule.fail -> error("")  
  | "/todos" result={Todo.resource} -> result  
  | "/user" result={User.resource} -> result  
  | "/login" result={User.resource} -> result  
  | "/admin" result={Admin.resource} -> result  
  | (.*) result={Todo.resource} -> result  

do Resource.register_external_js("/resources/js/google_analytics.js")  
server = Server.of_bundle([@static_resource_directory("resources")])  
server = Server.make(urls)

For a longer article/tutorial on dealing with external resources check out this blog post from the Opa team Dealing with External Resources.

Opa Database Migrations

Nicolas Glondu posted a comment on an earlier post detailing ways for doing database migrations with Opa. I thought it was useful enough that I should put up a post around it:

If you have complex changes in database structures in a OPA program, you have two choices :

1 – Keep both structure in your database and create a function which populates the new empty field from the other fields. Launch your program once with –db-force-upgrade and launch this function once (adding fields is safe but the new fields are empty). After this, clean removed fields and the function and re-run your program with –db-force-upgrade once more (removing fields is also safe). After this, you can keep using your program normally. A bit long but convenient for changes in small applications (with one instance running). You don’t have to relaunch your program immediately and you can spread ou migration on two program updates.

2 – Create another OPA program with both database definitions but with root nodes named differently [1]. Your program should take data from the original node and put it in the new node with your changes. That way you can have a standalone opa application for your migration. Which is more convenient if your program is used by other users. Once your program compiled, you can use command line options (–db-local) to use your true program’s database as the input root node an to put the output root node in a safe place. However, I don’t know if it works if your program has no specific root node defined.

[1] You can define a root node with database my_root = @meta and use it with db /my_root/toto : stringmap(int). You can have multiple root nodes in one program.