Rebar3 Features (part 6): _checkouts

In a build tool there is often a balancing act between ensuring repeatability and efficiency for the user. Wanting to make modifications on a dependency of your project is a common case of this. In rebar2 you could simply modify the source under deps/ and running rebar compile would pick those up. This meant that the contents of deps/ are not representative of the dependencies listed in rebar.config.

With rebar3 a dependency is never rebuilt, even if a source file changes. Before compiling changes to the project apps' source files rebar3 verifies that all dependencies are valid (meaning all artifacts exist and the .app file of the dependency matches the existing beam files for the dependency) and any that are not are fetch and/or built. To facilitate the workflow where a developer wants to modify one of these dependencies rebar3 introduces a feature call "checkout dependencies".

Checkout dependencies work by the user creating a directory _checkouts at the top level of the rebar3 project. Under _checkouts/ a copy of or symlink to an application's directory can be placed to have that copy take precedence over any dep in a config file, whether it has already been fetched and built or not. The apps under _checkouts are treated like project apps and any changes made to source files will be compiled. There is no need to delete any copy of the applications under _build/, rebar3 handles properly setting the code paths when building applications so only the version found under _checkouts will be used.

The checkouts also work for plugins. In the example below the structure of a _checkouts directory with two applications is shown. One being erlware_commons and the other a plugin rebar3_auto. This means that running rebar3 auto from this project will use the _checkouts copy of rebar3_auto and not one fetched to _build/default/plugins, a great way for performing manual testing on plugins you are developing.

$ tree _checkouts                 
├── erlware_commons -> ~/code/erlware_commons
└── rebar3_auto -> ~/code/rebar3_auto

Because the dependency used by the project is now not what is fetched and locked based on rebar.config or rebar.lock they will be removed from rebar.lock the next time compile is run. This makes clear to the developer that what is being built locally is not what would be built if pushed and fetched by another user, breaking repeatability. The developer must acknowledge this by making a point to commit the new lock file, if for whatever reason that is indeed a dependency to be removed.

Rebar3 Auto Compile and Load Plugin

During development the Erlang shell is often used for quickly testing functionality. Erlang's ability to reload modules while running makes this workflow even more efficient. To go a step further in removing manual intervention tools like sync and active have been created. These libs will listen for file modifications, recompile and reload the changed modules.

With rebar3 there is a plugin rebar3_auto which will start the shell, begin listening for modifications in the source directories of the project and recompile/reload when changes occur. Add to your global rebar3 configuration file to be able to use it on any of your projects ~/.config/rebar3/rebar.config:

{plugins, [rebar3_auto]}.

Running rebar3 auto will start the shell the same as running rebar3 shell but will be listening for file changes in your project's application source directories. When a file is change it will message the rebar3 agent to run compile and reload modules.

(master) $ rebar3 auto                       
===> Fetching rebar3_auto ({pkg,<<"rebar3_auto">>,<<"0.2.0">>})
===> Downloaded package, caching at /home/tristan/.cache/rebar3/hex/default/packages/rebar3_auto-0.2.0.tar
===> Fetching enotify ({pkg,<<"enotify">>,<<"0.1.0">>})
===> Version cached at ~/.cache/rebar3/hex/default/packages/enotify-0.1.0.tar is up to date, reusing it
===> Compiling enotify
===> Compiling rebar3_auto
Setting up watches.  Beware: since -r was given, this may take a while!  
Watches established.  
Erlang/OTP 18 [erts-7.1] [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V7.1  (abort with ^G)  
1> rlx_util:test().  
** exception error: undefined function rlx_util:test/0

Oops, our function doesn't exist. Switch to your editor and add to rlx_util.erl a function test/0 and export it, when you save you'll see it is recompiled and now available in the shell:

Verifying dependencies...  
Compiling relx

2> rlx_util:test().  

In the shell all the features of the shell command, its configuration and the rebar3 agent are available, see the docs for more information.

Rebar3 Features (part 5): Dependency Tracking

When starting rebar3 a goal was to never again end up in a situation where rm -rf deps to get the right set of dependencies was the quickest or only way forward.

An important piece of this is the new and improved upgrade command. However, this post will cover a less visible (it isn't a new command) improvement related to how rebar3 fetches and tracks dependencies each time the compile command is run.

Since there is no get-deps command, each time compile is run it relies on a task that verifies all dependencies are fetched and available in the appropriate build directory (specific to the profiles being used). Every resource type (pkg, git, hg and possibly third party types) implements the behaviour rebar_resource which includes the function needs_update/2. If the application is found to exist already during the dependency verification phase and is in the lock file rebar3 then uses needs_update/2 to check, unique to the resource type, if it is consistent with what is expected based on the lock file. If not, the source in the lock is fetched and replaces the current application.

An example project dep_tracking has been created to show what this looks like when switching branches in a project that has differing dependencies between the branches. In this example the master branch depends on cowboy 1.0.3 and a branch http2 depends on cowboy master where HTTP/2 support is being added. Just to show this doesn't change how transitive dependencies are handled, ranch has been added to the deps list as well, as a package 1.0.0 in master and 1.1.0 in http2.

Clone the repository and build as usual:

$ git clone
$ cd dep_tracking/
(master) $ cat rebar.lock
(master) $ rebar3 compile
 ===> Verifying dependencies...
===> Fetching cowboy ({git,"",{ref,"b8e4115eb13488c517d8d8ef33c47d0eaa7838c6"}})
===> Fetching ranch ({pkg,<<"ranch">>,<<"1.0.0">>})
===> Version cached at ~/.cache/rebar3/hex/default/packages/ranch-1.0.0.tar is up to date, reusing it
===> Fetching cowlib ({git,"",{ref,"d544a494af4dbc810fc9c15eaf5cc050cced1501"}})
===> Compiling cowlib
===> Compiling ranch
===> Compiling cowboy
===> Compiling dep_tracking

Notice that in the case of a package (a package to be specific) if a copy has already been fetched before, for any project, it is cached and will be reused instead of re-downloading, like in the case of git dependencies.

Now, even though all dependencies have been fetched and built, switching branches to the http2 branch and running compile again will fetch the new versions of the dependencies that need updating and compile them:

(master) $ git checkout http2
(http2) $ cat rebar.lock
(http2) $ rebar3 compile
===> Verifying dependencies...
===> Upgrading cowboy({git,"",{ref,"0ffde50991502ed222005376a66791debc4b991c"}})
===> Upgrading ranch ({pkg,<<"ranch">>,<<"1.1.0">>})
===> Version cached at ~/.cache/rebar3/hex/default/packages/ranch-1.1.0.tar is up to date, reusing it
===> Upgrading cowlib ({git,"",{ref,"14e597baa42a436469f99661ed7994685e269dc2"}})
===> Compiling cowlib
===> Compiling ranch
===> Compiling cowboy
===> Compiling dep_tracking

Rebar3 ensures the correct dependencies are fetched and built, simplifying working on different feature branches of your project or updating a branch with the upstream repository, without requiring any manual intervention.

May rm -rf deps be forever expunged from the vocabulary of free developers!

Rebar3 Features (part 4): Profiles

Running tests and need meck or proper? Building docs and want edown? Bundling up a target system and want to include erts and turn off relx's dev_mode? Rebar3 now has you covered for these scenarios through profiles.

Profiles can be named with any atom, can add new items to the configuration, or prepend new options to existing elements, and multiple profiles can be combined themselves.

The two special profiles are default and global. default is the profile everything is run under, with output going to _build/default/, unless another is specified in addition to default. When multiple profiles are used together the output directory is the profiles concatenated together with +, for example running rebar3 as test,prod <task> would produce _build/test+prod/, however the actual combination of profiles used in that run is default,test,prod, in the output default is always removed from the beginning unless it is the only profile in use.

The other special case for how profiles decide where output is written is global always refers to ~/.cache/rebar3/.

Providers are able to set profiles they will run under (in addition to default) with the {profiles, [atom()]} option to providers:create/1. Four providers that come with rebar3 specify a profile: eunit, ct and cover use test and edoc uses docs.

Examples of profile usage can help give an idea of how you might use them in your projects.

Since eunit, ct and cover run with the test profile adding deps specific to tests, like meck and eunit_formatters, which will be used when running rebar3 ct or any of the others, there is no need to include as test in the run, but to be clear, profiles are deduplicated so rebar3 as test ct will still be _build/test and not _build/test+test.

  [{test, [{deps,
             {eunit_formatters, {git, "git://", {branch, "master"}}}

            {eunit_opts, [
              {report, {eunit_progress, [colored, profile]}}

Another common dependency that in rebar2 would be included in the main dependency list and thus be fetched even when using used as a dependency is edown. With the docs profile that edoc runs with that is solved by moving edown under the profile:

{profiles, [{docs, 
            [{deps, [
              {edown, {git, "git://", {branch, "master"}}}

When developing a release, it is useful to use relx's dev_mode and to set include_erts to false. But when building a release for production you'll want the opposite. In this case, unlike with tests and docs, it is required to specify the profile you want to run the command with. Running rebar3 release will run as default, so with dev_mode true and include_erts false, while rebar3 as prod release pulls in the settings from the prod profile making dev_mode false and include_erts true.

{relx, [...
        {dev_mode, true},
        {include_erts, false},

  [{prod, [{relx, [
                   {dev_mode, false},
                   {include_erts, true}

The global profile is used in particular for plugins that the user defines in their personal rebar.config. For example I run rebar3 as global plugins upgrade to upgrade the two plugins in my ~/.config/rebar3/rebar.config:

{plugins, [rebar3_hex, rebar3_run]}.

Profiles are an important addition to the rebar configuration for making development and dependency management simpler. Please be a good Erlang citizen and separate your dependencies into the appropriate profiles, those who depend on your application will appreciate it.

Rebar3 features (part 3): Overrides

What do you do when a dependency has settings in its rebar.config that are causing you problems? Maybe it includes dependencies that are not needed in the general case, like meck or edown. Or it could have set a required OTP version that isn't accurate and you want to remove. Or the app could contain C code that needs compiling and had relied on rebar2's port compiler. These problems often lead to forks of projects, which isn't good for anyone, so in rebar3 we've added a feature called overrides.

Overrides allow any rebar.config at a higher level than a dependency to either replace or add to the configuration of all or an individual application at a lower level.

The type spec for overrides looks like:

{overrides, [{add, atom(), [{atom(), any()]}
             | {override, atom(), [{atom(), any()]},
             | {override, [{atom(), any()]} 

The bitcask application from Basho is configured to be built with rebar2, but instead of forking the project and patching the config, a user can instead simply add the below overrides section to their rebar.config:

    {override, bitcask,
      [{deps, []},
       {plugins, [pc]},
       {artifacts, ["priv/"]},
       {provider_hooks, [{post, 
                          [{compile, {pc, compile}},
                           {clean, {pc, clean}}]

These overrides for bitcask replace its deps entry with an empty list, this removes the meck dependency which is only needed if you were running tests and the cuttlefish dependency which isn't required. Plus, since the port compiler functionality was removed in rebar3, the port compiler plugin must be added and hooked in to compile and clean with provider_hooks.

The rest of the guts of the override will be covered in future posts, see the docs for more information on provider_hooks, artifacts and plugins