rebar3

Rebar3 features (part 3): Overrides

What do you do when a dependency has settings in its rebar.config that are causing you problems? Maybe it includes dependencies that are not needed in the general case, like meck or edown. Or it could have set a required OTP version that isn't accurate and you want to remove. Or the app could contain C code that needs compiling and had relied on rebar2's port compiler. These problems often lead to forks of projects, which isn't good for anyone, so in rebar3 we've added a feature called overrides.

Overrides allow any rebar.config at a higher level than a dependency to either replace or add to the configuration of all or an individual application at a lower level.

The type spec for overrides looks like:

{overrides, [{add, atom(), [{atom(), any()]}
             | {override, atom(), [{atom(), any()]},
             | {override, [{atom(), any()]} 
            ]}.

The bitcask application from Basho is configured to be built with rebar2, but instead of forking the project and patching the config, a user can instead simply add the below overrides section to their rebar.config:

{overrides, 
  [
    {override, bitcask,
      [{deps, []},
       {plugins, [pc]},
       {artifacts, ["priv/bitcask.so"]},
       {provider_hooks, [{post, 
                          [{compile, {pc, compile}},
                           {clean, {pc, clean}}]
                         }]}
      ]
    }
  ]
}.

These overrides for bitcask replace its deps entry with an empty list, this removes the meck dependency which is only needed if you were running tests and the cuttlefish dependency which isn't required. Plus, since the port compiler functionality was removed in rebar3, the port compiler plugin must be added and hooked in to compile and clean with provider_hooks.

The rest of the guts of the override will be covered in future posts, see the docs for more information on provider_hooks, artifacts and plugins

Rebar3 Features (part 2): Dependency Tree

rebar3 tree is a new command to allow the user to view what dependency pulled in each transitive dependency. This is especially useful with rebar3's dependency resolution strategy of "first wins".

Thanks for pushing for this feature goes to Heinz N. Gies and inspiration comes from leiningen's command lein deps :tree.

For an example I've cloned chef-server and built oc_erchef under src/oc_erchef. It is unique because it has both a top level app oc_erchef under src/ as well as additional project apps under apps/.

Additionally, I've added _checkouts/erlware_commons to show how checkout dependencies, an application linked to under _checkouts/, is moved to the top level and marked as (checkout app) and switched lager to a hex package dependency, the rest being git sourced dependencies.

$ rebar3 tree
├─ chef_db─12.1.2-6047c67 (project app)
├─ chef_index─12.1.2-6047c67 (project app)
├─ chef_objects─12.1.2-6047c67 (project app)
├─ chef_test─12.1.2-6047c67 (project app)
├─ depsolver─12.1.2-6047c67 (project app)
├─ erlware_commons─0.16.0 (checkout app)
├─ oc_chef_authz─12.1.2-6047c67 (project app)
├─ oc_chef_wm─12.1.2-6047c67 (project app)
└─ oc_erchef─12.1.2-6047c67 (project app)
   ├─ bcrypt─0.0.0+build.87.ref085eb59 (git repo)
   ├─ chef_authn─0.0.0+build.86.refe7850d0 (git repo)
   ├─ darklaunch─0.0.0+build.72.ref05881cb (git repo)
   │  └─ meck─0.8.3 (git repo)
   ├─ efast_xs─0.1.0 (git repo)
   ├─ ej─0.0.0+build.87.ref132a9a3 (git repo)
   ├─ envy─0.0.0+build.38.ref954c87a (git repo)
   ├─ eper─0.90.0 (git repo)
   ├─ folsom─0.0.0+build.335.ref38e2cce (git repo)
   │  └─ bear─0.0.0+build.32.ref1192345 (git repo)
   ├─ folsom_graphite─0.0.0+build.41.refd4ce9bf (git repo)
   ├─ gen_bunny─0.1 (git repo)
   │  ├─ amqp_client─0.0.0 (git repo)
   │  └─ rabbit_common─0.0.0 (git repo)
   │     └─ gen_server2─1.0.0 (git repo)
   ├─ ibrowse─4.0.1.1 (git repo)
   ├─ jiffy─0.0.0+build.131.reff661ee9 (git repo)
   ├─ lager─2.1.1 (hex package)
   │  └─ goldrush─0.1.6 (hex package)
   ├─ mini_s3─0.0.1 (git repo)
   ├─ mixer─0.1.1 (git repo)
   ├─ neotoma─0.0.0+build.125.ref760928e (git repo)
   ├─ opscoderl_folsom─0.0.1 (git repo)
   ├─ opscoderl_httpc─0.0.1 (git repo)
   ├─ opscoderl_wm─0.0.1 (git repo)
   │  └─ webmachine─0.0.0+build.526.ref7677c24 (git repo)
   │     └─ mochiweb─2.9.0 (git repo)
   ├─ pooler─0.0.0+build.159.ref7bb8ab8 (git repo)
   ├─ sqerl─1.0.0 (git repo)
   │  └─ epgsql─3.1.0 (git repo)
   ├─ stats_hero─0.0.0+build.73.refff00041 (git repo)
   │  └─ edown─0.2.4+build.66.ref30a9f78 (git repo)
   ├─ sync─0.1.3 (git repo)
   └─ uuid─1.3.2 (git repo)
      └─ quickrand─1.3.2 (git repo)

Rebar3 Features (part 1): Local install and upgrade

Update January 22, 2016: The original post has been modified to reflect that the commands install and upgrade were moved from the unstable namespace to the local namespace. Also added is use of the environment variable $REBAR3_ERL_ARGS in the run script which allows the user to specify custom Erlang VM arguments like: REBAR3_ERL_ARGS="+A10" rebar3 shell.

Rebar3 comes with a lot of new and improved features. I'll be publishing posts here to highlight some of these features over the coming weeks.

Installing and Upgrading Rebar3

Rebar is an escript bundle, and this has been very important to its ease of use, mainly because they let Erlang users have a single file acting as an executable, regardless of the underlying Erlang installation, which can even be committed to a project's repository.

So, rebar3 is also an escript. Pre-built escripts can be downloaded from s3:

$ wget https://s3.amazonaws.com/rebar3/rebar3

or clone and bootstrap the git repo:

$ git clone https://github.com/rebar/rebar3.git
$ cd rebar3
$ ./bootstrap

However, escripts do have their drawbacks. They are slower to start, rely on the old Erlang io server, making rebar3 shell not act exactly the same as an erl shell, and it can lead to repos storing the escript along with the code but never upgrading it.

So in rebar3 we've introduced the ability to extract the escript archive along with a run script to ~/.cache/rebar3/, plus a command that will fetch and do the same for the latest escript release of rebar3 from s3.

The install command will be under a provider namespace, originally it lived under unstable, the namespace for experimental features that are likely to change in the near future and that aren't yet considered stable, but now is in the local namespace:

$ ./rebar3 local install
===> Extracting rebar3 libs to $HOME/.cache/rebar3/lib...
===> Writing rebar3 run script $HOME/.cache/rebar3/bin/rebar3...
===> Add to $PATH for use: export PATH=$HOME/.cache/rebar3/bin:$PATH

Follow the instructions for adding the rebar3 bin directory to your $PATH, and optionally add it to your shell's configuration, such as ~/.bashrc or ~/.zshrc.

To upgrade rebar3 use rebar3 unstable upgrade which will fetch the latest escript and extract it:

$ rebar3 local upgrade

We hope this new method for installing, upgrading and running rebar3 will allow people to be comfortable with not bundling rebar3 in their project's repositories, resolve any compilation speed differences between rebar3 and alternative Erlang build tools and help keep everyone up to date with the latest features and bug fixes.