Automatic Hex Package Publishing with Travis-CI

Note: Travis CI does not yet provide rebar3 as a build tool for Erlang projects, so you will have to include a rebar3 executable escript in your repository.

In your application's .app.src file use "git" as the version. This will replace the version string with the current tag when publishing the hex package:


In the .travis.yml configuration add the below section so that the script scripts/ is run everytime a new tag is made:

  provider: script
  script: scripts/
    tags: true

Create the script under scripts/ and make it executable:

#!/usr/bin/env bash

## Setup hex user
mkdir -p ~/.hex  
echo '{username,<<"'${HEX_USERNAME}'">>}.' > ~/.hex/hex.config  
echo '{key,<<"'${HEX_KEY}'">>}.' >> ~/.hex/hex.config

## Add the rebar3 hex plugin to global
mkdir -p ~/.config/rebar3  
echo '{plugins, [rebar3_hex]}.' > ~/.config/rebar3/rebar.config

./rebar3 hex publish <<EOF

In the script it relies on two environmental variables $HEX_USERNAME and $HEX_KEY. Using either the Travis CLI tool or web interface you can create these variables and have the values encrypted or unseen by viewers of your repository. For instructions see the Travis-CI environment variables docs.

Rebar3 Hex Plugin

No plugin is needed for using Hex packages in your project, covered in documentation here, but at this time the plugin rebar3_hex is needed for publishing. The plugin provides support for registering with and publishing packages. For instructions on registering see the rebar3 documentation, here we'll cover publishing a package with some unique features to the rebar3 plugin.

rebar3 hex cut

The cut command provides some automation on top of publish. The rebar3 hex plugin itself is a good example of a use case for this command. The .app.src file for the plugin looks like:

$ cat src/ 
{application, rebar3_hex,
  [{description, " plugin for rebar3"},
   {vsn, "git"},
   {applications, [kernel, stdlib, jsx]},
   {contributors, ["Tristan Sloughter"]},
   {licenses, ["MIT"]},
   {links, [{"Github", ""}]}]}.

contributors, licenses, links are hex specific metadata. For the vsn of the application a commonly used feature of rebar3 is used where "git" is in the .app.src file and replaced at compile time with the latest git tag, if the current HEAD isn't on a specific tag the last tag with appended build metadata, as documented on, is used.

Running cut will first ask for what increment of the version to perform, major, minor or patch. The version, based on the git tag, is then incremented and a new tag is created. After publishing the package cut asks if the git tag should be pushed to origin.

In the case an application has a static version string (not one like "git") cut will increment the version and update .app.src and create a version bump commit and push to origin.

Below is the output from publishing a version of rebar3_hex with the cut command:

(master) $ rebar3 hex cut             
Select semver increment or other (Current 1.7.1):  
1) patch  
2) minor  
3) major  
4) other  
[1-4] > 1
===> Creating new tag v1.7.2...
Publishing rebar3_hex 1.7.2  
    jsx 2.6.1
  Excluded dependencies (not part of the Hex package):

  Included files:
Proceed? ("Y")> Y  
===> Published rebar3_hex 1.7.2
Push new tag to origin? ("Y")> Y  
===> Pushing new tag v1.7.2...

Publishing from an umbrella project

In an umbrella project there are multiple applications in the project. Since a package must be a single application the entire project can not be published as one. The rebar3 hex plugin will ask which application, or all, in the umbrella the user wants to publish.

Remember to update the local rebar3 package registry with rebar3 update to fetch the latest published packages.

With and rebar3 hex an efficient and effective Erlang package ecosystem can grow.

Rebar3 Features (part 6): _checkouts

In a build tool there is often a balancing act between ensuring repeatability and efficiency for the user. Wanting to make modifications on a dependency of your project is a common case of this. In rebar2 you could simply modify the source under deps/ and running rebar compile would pick those up. This meant that the contents of deps/ are not representative of the dependencies listed in rebar.config.

With rebar3 a dependency is never rebuilt, even if a source file changes. Before compiling changes to the project apps' source files rebar3 verifies that all dependencies are valid (meaning all artifacts exist and the .app file of the dependency matches the existing beam files for the dependency) and any that are not are fetch and/or built. To facilitate the workflow where a developer wants to modify one of these dependencies rebar3 introduces a feature call "checkout dependencies".

Checkout dependencies work by the user creating a directory _checkouts at the top level of the rebar3 project. Under _checkouts/ a copy of or symlink to an application's directory can be placed to have that copy take precedence over any dep in a config file, whether it has already been fetched and built or not. The apps under _checkouts are treated like project apps and any changes made to source files will be compiled. There is no need to delete any copy of the applications under _build/, rebar3 handles properly setting the code paths when building applications so only the version found under _checkouts will be used.

The checkouts also work for plugins. In the example below the structure of a _checkouts directory with two applications is shown. One being erlware_commons and the other a plugin rebar3_auto. This means that running rebar3 auto from this project will use the _checkouts copy of rebar3_auto and not one fetched to _build/default/plugins, a great way for performing manual testing on plugins you are developing.

$ tree _checkouts                 
├── erlware_commons -> ~/code/erlware_commons
└── rebar3_auto -> ~/code/rebar3_auto

Because the dependency used by the project is now not what is fetched and locked based on rebar.config or rebar.lock they will be removed from rebar.lock the next time compile is run. This makes clear to the developer that what is being built locally is not what would be built if pushed and fetched by another user, breaking repeatability. The developer must acknowledge this by making a point to commit the new lock file, if for whatever reason that is indeed a dependency to be removed.

Rebar3 Auto Compile and Load Plugin

During development the Erlang shell is often used for quickly testing functionality. Erlang's ability to reload modules while running makes this workflow even more efficient. To go a step further in removing manual intervention tools like sync and active have been created. These libs will listen for file modifications, recompile and reload the changed modules.

With rebar3 there is a plugin rebar3_auto which will start the shell, begin listening for modifications in the source directories of the project and recompile/reload when changes occur. Add to your global rebar3 configuration file to be able to use it on any of your projects ~/.config/rebar3/rebar.config:

{plugins, [rebar3_auto]}.

Running rebar3 auto will start the shell the same as running rebar3 shell but will be listening for file changes in your project's application source directories. When a file is change it will message the rebar3 agent to run compile and reload modules.

(master) $ rebar3 auto                       
===> Fetching rebar3_auto ({pkg,<<"rebar3_auto">>,<<"0.2.0">>})
===> Downloaded package, caching at /home/tristan/.cache/rebar3/hex/default/packages/rebar3_auto-0.2.0.tar
===> Fetching enotify ({pkg,<<"enotify">>,<<"0.1.0">>})
===> Version cached at ~/.cache/rebar3/hex/default/packages/enotify-0.1.0.tar is up to date, reusing it
===> Compiling enotify
===> Compiling rebar3_auto
Setting up watches.  Beware: since -r was given, this may take a while!  
Watches established.  
Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V7.1  (abort with ^G)  
1> rlx_util:test().  
** exception error: undefined function rlx_util:test/0

Oops, our function doesn't exist. Switch to your editor and add to rlx_util.erl a function test/0 and export it, when you save you'll see it is recompiled and now available in the shell:

Verifying dependencies...  
Compiling relx

2> rlx_util:test().  

In the shell all the features of the shell command, its configuration and the rebar3 agent are available, see the docs for more information.

Rebar3 Features (part 5): Dependency Tracking

When starting rebar3 a goal was to never again end up in a situation where rm -rf deps to get the right set of dependencies was the quickest or only way forward.

An important piece of this is the new and improved upgrade command. However, this post will cover a less visible (it isn't a new command) improvement related to how rebar3 fetches and tracks dependencies each time the compile command is run.

Since there is no get-deps command, each time compile is run it relies on a task that verifies all dependencies are fetched and available in the appropriate build directory (specific to the profiles being used). Every resource type (pkg, git, hg and possibly third party types) implements the behaviour rebar_resource which includes the function needs_update/2. If the application is found to exist already during the dependency verification phase and is in the lock file rebar3 then uses needs_update/2 to check, unique to the resource type, if it is consistent with what is expected based on the lock file. If not, the source in the lock is fetched and replaces the current application.

An example project dep_tracking has been created to show what this looks like when switching branches in a project that has differing dependencies between the branches. In this example the master branch depends on cowboy 1.0.3 and a branch http2 depends on cowboy master where HTTP/2 support is being added. Just to show this doesn't change how transitive dependencies are handled, ranch has been added to the deps list as well, as a package 1.0.0 in master and 1.1.0 in http2.

Clone the repository and build as usual:

$ git clone
$ cd dep_tracking/
(master) $ cat rebar.lock
(master) $ rebar3 compile
 ===> Verifying dependencies...
===> Fetching cowboy ({git,"",{ref,"b8e4115eb13488c517d8d8ef33c47d0eaa7838c6"}})
===> Fetching ranch ({pkg,<<"ranch">>,<<"1.0.0">>})
===> Version cached at ~/.cache/rebar3/hex/default/packages/ranch-1.0.0.tar is up to date, reusing it
===> Fetching cowlib ({git,"",{ref,"d544a494af4dbc810fc9c15eaf5cc050cced1501"}})
===> Compiling cowlib
===> Compiling ranch
===> Compiling cowboy
===> Compiling dep_tracking

Notice that in the case of a package (a package to be specific) if a copy has already been fetched before, for any project, it is cached and will be reused instead of re-downloading, like in the case of git dependencies.

Now, even though all dependencies have been fetched and built, switching branches to the http2 branch and running compile again will fetch the new versions of the dependencies that need updating and compile them:

(master) $ git checkout http2
(http2) $ cat rebar.lock
(http2) $ rebar3 compile
===> Verifying dependencies...
===> Upgrading cowboy({git,"",{ref,"0ffde50991502ed222005376a66791debc4b991c"}})
===> Upgrading ranch ({pkg,<<"ranch">>,<<"1.1.0">>})
===> Version cached at ~/.cache/rebar3/hex/default/packages/ranch-1.1.0.tar is up to date, reusing it
===> Upgrading cowlib ({git,"",{ref,"14e597baa42a436469f99661ed7994685e269dc2"}})
===> Compiling cowlib
===> Compiling ranch
===> Compiling cowboy
===> Compiling dep_tracking

Rebar3 ensures the correct dependencies are fetched and built, simplifying working on different feature branches of your project or updating a branch with the upstream repository, without requiring any manual intervention.

May rm -rf deps be forever expunged from the vocabulary of free developers!