Projmake-mode: Flymake Replacement

There is a great new Emacs plugin from Eric Merritt that like FlyMake builds your code and highlights within Emacs any errors or warnings, but unlike FlyMake builds across the whole project. You can clone the mode from here projmake-mode

After cloning the repo to your desired location add this bit to your dot emacs file, replacing <PATH> with the path to where you cloned the repo.

This Emacs code also uses add-hook to set projmake-mode to start for erlang-mode is loaded. Projmake by default knows how to handle rebar and Make based builds so there is no setup after this, assuming your project is built this way.

Here is my Makefile for building Erlang projects with rebar, replace PROJECT with the name of your project:

Now you can load Emacs and a file from your project and if it is an Erlang file due to the add-hook function in our dot emacs file it will automatically load projmake-mode. You can add hooks for other modes or simply run M-x projmake-mode.

For more documentation and how to extend to other types of projects check out the documentation.

Opa Plugin Development with a PubNub Example

This will be a two part series of posts on writing plugins for Opa. One of Opa’s greatest features is you write everything in the functional, staticly typed Opa language. This even includes the frontend code you’d usually do in Javascript. This means your code is less error-prone which significantly reduces the amount of time spent on debugging code.

But how do you use Javascript library X within your Opa code?

While you can include Javascript code in a resource, like we do here for Google Analytics:

do Resource.register_external_js("/resources/js/google_analytics.js")

This isn’t what we want to do if we want to call external Javascript functions from Opa code. For this we create an Opa plugin. In this post I’ll be showing how I built the Opa client side bindings for the PubNub Javascript API, fork it on github and see all the other language bindings. PubNub is a cloud hosted real-time publish and subscribe interface for all types of applications.

Opa allows binding to external functions through its Binding System Library (BSL). In order to use this to create a plugin for some external Javascript, we create a file pubnub.js that we will compile to a plugin using opa-plugin-builder. Below is an example of one of the external functions that needs to be defined to use PubNub’s publish.

##register publish : string, string -> void
##args(c, m)
{
    PUBNUB.publish({
                       channel : c, message : m
                   });

    return js_void;
}

The first line registers the function publish and specifies its arguments to be two strings and its return value to be void. The next line names the arguments. You can think of ##args like Javascript’s function keyword. In the function, we can use the PUBNUB library just like we would normally. Here we pass to publish the channel and the message to be published. Note that the message can be any Javascript object, even though its a string to the plugin. We simply must serialize the JSON before passing to publish, which we’ll see shortly.

Now after defining external functions for all PubNub api calls in the pubnub.js file we can compile it:

$ opa-plugin-builder src/plugin/pubnub.js -o pubnub

This creates the plugin pubnub.opp.

To make using the plugin easier, we create a package in pubnub.opa that will contain a module PubNub. Below is the contents of pubnub.opa for the publish function that will show how to register the external functions and create a module named PubNub around them.

package pubnub

pubnub_publish = %%pubnub.publish%%

@client publish_client = pubnub_publish

PubNub = {{
  publish(channel: string, j : RPC.Json.json) =
    publish_client(channel, Json.to_string(j))
}}

do Resource.register_external_js("http://cdn.pubnub.com/pubnub-3.1.min.js")

The first thing to notice is setting the pubnub_publish variable. The %% syntax is specifying that pubnub.publish is the key of an external primitive. pubnub is the name of the plugin (pubnub.opp) and publish is the function we registered in the plugin.

After this we create another variable publish_client that specifies that that this function (variables can contain functions) is to be run on the client side. Lastly, we define a module PubNub and a publish function inside the module to be used by our Opa programs.

At the end we make sure that the needed PubNub Javascript file is includes in all resources by registering it and giving the URL to the PubNub CDN.

Now we can compile it all together with the opa-plugin-builder command from above and use the plugin.

import pubnub

broadcast(author, msg) =
   username : RPC.Json.json = {String = author}
   message : RPC.Json.json = {String = msg}
   record_json : RPC.Json.json = {Record = [("username", username), ("message", message)]}
   do PubNub.publish("chat", record_json)
   Dom.clear_value(#entry)

The broadcast function is part of pubnub_chat.opa. broadcast takes 2 arguments, the username of the user sending the message and the message itself. We then construct three variables of Opa’s json type, username, message and record_json. username and message are simply Json strings and they are then combined into a Json Record type that is a list of string/Json pairs. Since our PubNub.publish function converts the Opa Json to a string before sending to the Javascript binding we can simply pass the Opa Json variable record_json.

Lastly, lets take a quick look at the subscribe function. Defining it in pubnub.js look like:

##register subscribe : string, (string -> void) -> void

Since it takes a callback function you’ll notice that the second argument’s type looks different from in publish. (string -> void) is defining that the function takes a variable that is a function which takes a single argument of type string and has a void return type.

Now in pubnub.opa we have a subscribe function that is hardly different from the publish function we described above.

subscribe(channel: string, callback) =
  subscribe_client(channel, callback)

But how do we define a function we can pass to subscribe? That is the great thing, it can be any Opa function!

PubNub.subscribe("chat", (x -> user_update(x)))

Above we are subscribing to the channel chat and passing an anonymous function that takes a single argument and sends it to the function user_update.

user_update(x) =
  match Json.deserialize(x) with
         | {some={Record = [("username", username), ("message", message)]}} ->
                    line = <div class="line"> <div class="user">{Json.to_string(username)}:</>
                             <div class="message">{Json.to_string(message)}</>
                            </>
                    do Dom.transform([#conversation +<- line ])
                    Dom.scroll_to_bottom(#conversation)
         | _ -> void

Now every time a message is posted to the chat channel, our callback, the anonymous function, will be run and call the function user_update which we see deserializes the Json string to get an Opa Json Record. The rest is basic Opa DOM manipulation to add the new message to the messages in the conversation element.

In my next post on Opa and PubNub, I’ll be describing an Opa server side API for PubNub.

Major OpaDo Speed-Up with @publish

I received a patch for OpaDo from Francois Regis Sinot that has made the hosted OpaDo http://opado.org on Dotcloud MUCH faster, adding and removing item is now instantaneous. He placed @publish around explicitly server side functions that caused adding todo items to go from 21 to 1 server calls. I thought this simple but maybe not obvious for new Opa developers (it certainly wasn’t to me!) needed a blog post so that beginners like my self may find it and be able to start using the @publish directive. Check out the Opa manual for some more information on @publish, specifically I’d look at Section 4.7.4 and 10.2

Below are the changes he made. Which you can also of see in the commit diff on Github and check out the entire source.

    
-   @async
-   db_make_done(username: string, id: string) =
+   @publish @async
+   db_make_done(id: string) =
+       username = User.get_username()
        items = /todo_items[username]
        item = Option.get(StringMap.get(id, items))
        /todo_items[username] <- StringMap.add(id, {item with done=true}, items)
 
-   @async
+   @publish @async
    db_remove_item(id: string) =
      username = User.get_username()
      items = /todo_items[username]
 
-   @async
+   @publish @async
    db_add_todo(id: string, x: string) =
      username = User.get_username()
      items = /todo_items[username]
      /todo_items[username] <- StringMap.add(id, { value=x done=false created_at="" }, items)
 
+   @publish
    add_todos() =
      username = User.get_username()
      items = /todo_items[username]

Opa Database Migrations

Nicolas Glondu posted a comment on an earlier post detailing ways for doing database migrations with Opa. I thought it was useful enough that I should put up a post around it:

If you have complex changes in database structures in a OPA program, you have two choices :

1 – Keep both structure in your database and create a function which populates the new empty field from the other fields. Launch your program once with –db-force-upgrade and launch this function once (adding fields is safe but the new fields are empty). After this, clean removed fields and the function and re-run your program with –db-force-upgrade once more (removing fields is also safe). After this, you can keep using your program normally. A bit long but convenient for changes in small applications (with one instance running). You don’t have to relaunch your program immediately and you can spread ou migration on two program updates.

2 – Create another OPA program with both database definitions but with root nodes named differently [1]. Your program should take data from the original node and put it in the new node with your changes. That way you can have a standalone opa application for your migration. Which is more convenient if your program is used by other users. Once your program compiled, you can use command line options (–db-local) to use your true program’s database as the input root node an to put the output root node in a safe place. However, I don’t know if it works if your program has no specific root node defined.

[1] You can define a root node with `database my_root = @meta` and use it with `db /my_root/toto : stringmap(int)`. You can have multiple root nodes in one program.

OpaDo: Personal ToDo Lists

This is a continuation of two past posts (one, two) on my first application with Opa called OpaDo. You can try the live demo here and check out the full source code on Github

Updating OpaDo to add user accounts the project structure has been changed a bit and modularized. Below is the new project layout.

opado/
├── Makefile
├── README.md
├── dotcloud.yml
├── resources
│   ├── destroy.png
│   └── todos.css
└── src
    ├── main.opa
    ├── todo.opa
    └── user.opa

Now there is a main, todo and user module. The main module is the entry point for the app and looks like:

package opado.main

import opado.user
import opado.todo

urls : Parser.general_parser(http_request -> resource) =
  parser
  | {Rule.debug_parse_string(s -> Log.notice("URL",s))} Rule.fail -> error("")
  | "/todos" result={Todo.resource} -> result
  | "/user" result={User.resource} -> result
  | "/login" result={User.resource} -> result
  | (.*) result={Todo.resource} -> result

server = Server.of_bundle([@static_resource_directory("resources")])
server = Server.make(urls)

Here we define the name of this package and import the user and todo modules. Next is the url matching code. urls is a parser that takes an HTTP request and returns a resource. The matching is pretty straight forward. For example:

  | "/todos" result={Todo.resource} -> result

Here we are matching on URLs that begin with /todos but could have anything after that. What is contained after /todos is passed to the Todo.resource which the variable result is set to. And finally that result is returned.

The last two lines simple define the reource directory for the server and pass in the matching function for the HTTP requests.

The todo resource isn’t important to us in this post since its hardly changed. But there are a two important changes:

db /todo_items : stringmap(stringmap(todo_item))
db /todo_items[_][_]/done = false

Here we see that the /todo_items database is not longer simply a stringmap of todo_item‘s but a stringmap of a that. This is so we can reference the items by a user identifier. For example a user identified by the string “user01″ who has a todo item identified by “aaa” would be read from the data base as /todo["user01"]["aaa"].

There are a few other changes to the todo module so that items are properly inserted for the logged in user and deleting must be done in the second stringmap. But we’ll move on to the user module now.

Much of the user module was taken from Matthieu Guffroy’s OpaCMS code on github. But I’ve made a number of modification for my needs.

@abstract type User.password = string
@abstract type User.ref = string

type User.t =
  {
    username : string
    fullname : string
    password : User.password
  }

type User.status = { logged : User.ref } / { unlogged }
type User.info = UserContext.t(User.status)
type User.map('a) = ordered_map(User.ref, 'a, String.order)

db /users : User.map(User.t)

User_data = {{
  mk_ref( login : string ) : User.ref =
    String.to_lower(login)

  ref_to_string( login : User.ref ) : string =
    login

  save( ref : User.ref, user : User.t ) : void =
    /users[ref] <- user

  get( ref : User.ref ) : option(User.t) =
    ?/users[ref]
}}

Above we have the data, types and database definitions necessary to handle the users.

User.t provides the record for storing necessary user data. Next, we have types for checking the user status of if they are logged in or not.

UserContext is a module provided by Opa for dealing with associating the user values with the client — via cookies. And the data for that user can only accessed by the user that owns it.

User_data object provides functions for accessing and manipulating users.

Now we can look at the User module.

User = {{

  @private state = UserContext.make({ unlogged } : User.status)

  create(username, password) =
    do match ?/users[username] with
      | {none} ->
          user : User.t =
            { username=username ;
              fullname="" ;
              password = Crypto.Hash.sha2(password) }
          /users[username] <- user

      | _ -> void
    Client.goto("/login")

At the beginning of the User object we declare a UserContext and a function for creating new users. The function simply checks if the user exists already with the match statement and if not creates a new User.t record and inserts it to the users database.

If we wish to login we must also modify the UserContext

  login(login, password) =
    useref = User_data.mk_ref(login)
    user = User_data.get(useref)
    do match user with
     | {some = u} -> if u.password == Crypto.Hash.sha2(password) then
                       UserContext.change(( _ -> { logged = User_data.mk_ref(login) }), state)
     | _ -> void
    Client.goto("/todos")

The function attempts to read the user from the database and checks if the passwords match. If so, it will set the UserContext to logged in. The function then tells the client to go to /todos. If the login was unsuccessful, it doesn’t matter and will just redirect to the sign up page.

Obviously, better error handling and notification is the next step for the application.

The last interesting part for this I think is the request matching. The rest of the code is mostly just HTML and piecing together the functions I already described.

  resource : Parser.general_parser(http_request -> resource) =
    parser
    | "/new" ->
      _req -> Resource.styled_page("New User", ["/resources/todos.css"], new())
    | "/edit" ->
      _req -> edit()
    | "/view/" login=(.*) ->
      _req -> view(Text.to_string(login))
    | .* ->
      _req -> start()

The key match to look at is:

    | "/view/" login=(.*) ->
      _req -> view(Text.to_string(login))

This shows the request matching /view, which in this case comes after the main module matches ‘/user’ and routes to the User module resource. But then we have login=(.*), this is matching the variable login to the rest of the url. This variable login can then be used in view(Text.to_string(login)) to pass to the view function so it knows what user is being asked to be displayed.

There’ll be more to come. Next, I need to add some validation, an admin page and then the ability for users to have categories to organize their todo items under.

And let me know anything else people would like to see!