Introduction

RabbitMQ streams is built on top of osiris. osiris models an append-only persistent log and I though it would be interesting to see if it would be possible to use it outside of RabbitMQ.

The problem

osiris is not available through hex.pm so we include it using the github directive:

# mix.exs
defp deps do
  {:osiris, github: "rabbitmq/osiris"}
end

But compiling results in an error

mix deps.get
mix deps.compile

===> Compilation failed: there is no code in this directory (/$HOME/dev/deps_test/_build/dev/lib/osiris), it is unreadable or for some other reason is not a recognizable application structure.
** (Mix) Could not compile dependency :osiris, "/$HOME/.asdf/installs/elixir/1.13.1-otp-24/.mix/rebar3 bare compile --paths /$HOME/deps_test/_build/dev/lib/*/ebin" command failed. Err

This is likely due to mix interpreting osiris as rebar3 project due to the rebar.config in the root of the project. By specifying the :manger option to use :make the automatic project type inference is bypassed:

# mix.exs
defp deps do
  {:osiris, github: "rabbitmq/osiris", manager: :make}
end

So now it compiles but if we try run the application the following happens:

mix
** (Mix) Could not start application gen_batch_server: could not find application file: gen_batch_server.app

When osiris is compiled the dependencies gen_batch_server and seshat are fetched. However the dependencies are not added to the deps directory but is found in deps/osiris/deps. We can fix this by adding the dependencies to the project, since they’re both available through hex:

  defp deps do
    [
      {:osiris, github: "rabbitmq/osiris", manager: :make},
      # Osiris deps,
      {:gen_batch_server, "~> 0.8.8", override: true},
      {:seshat, "~> 0.3.2", override: true},
    ]
  end

Another approach is to include the dependencies via the :path directive by resolving them ourself:

  def project do
    [
      app: :deps_test,
      version: "0.1.0",
      elixir: "~> 1.13",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      aliases: aliases()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger]
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    deps = [
      {:osiris, github: "rabbitmq/osiris", manager: :make}
    ]

    nested_deps = resolve_deps(:osiris, [:gen_batch_server, :seshat], Mix.Project.deps_path())
    deps ++ nested_deps
  end

  defp mk_deps(_) do
    path = Path.join([Mix.Project.deps_path(), "osiris"])
    File.cd!(path, fn ->
      System.cmd("make", ["fetch-deps"])
    end)
  end

  defp resolve_deps(name, deps, deps_path) do
    base_path = Path.join([deps_path, to_string(name)])

    for dep <- deps do
      build_dep(dep, base_path)
    end
  end

  defp build_dep(name, path) do
    {
      name,
      path: Path.join([path, "deps", to_string(name)]),
    }
  end

  defp aliases do
    [
      deps_all: [
        "deps.get",
        &mk_deps/1
      ]
    ]
  end

mix deps.get fetches osiris but its dependencies are not fetched until mix deps.compile is called. (since deps.compile will run make in deps/osiris).

We define the alias deps_all which calls mix deps.get and then calls make fetch-deps from osiris making them available on the paths previously defined using resolve_deps/3.

Another slightly more automatic but equally “hacky” approach would be to a check in deps/0 if the osiris directory exists so that we can call make list-deps and create path dependencies for all dependencies listed:

defmodule DepsTest.MixProject do
  use Mix.Project

  def project do
    [
      app: :deps_test,
      version: "0.1.0",
      elixir: "~> 1.13",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      aliases: aliases()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger]
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    deps = [
      {:osiris, github: "rabbitmq/osiris", manager: :make}
    ]

    nested_deps = resolve_deps(:osiris, Mix.Project.deps_path())
    deps ++ nested_deps
  end

  defp mk_deps(_) do
    path = Path.join([Mix.Project.deps_path(), "osiris"])
    File.cd!(path, fn ->
      System.cmd("make", ["fetch-deps"])
    end)
  end

  defp resolve_deps(name, deps_path) do
    base_path = Path.join([deps_path, to_string(name)])

    if File.exists?(base_path) do
      File.cd!(base_path, fn ->
        case System.cmd("make", ["list-deps"]) do
          {deps, 0} ->
            deps
            |> String.split("\n", trim: true)
            |> Enum.map(&build_dep/1)
          {out, code} ->
            raise "`make list-deps` error: #{inspect(out)} code: #{code}"
        end
      end)
    else
      []
    end
  end

  defp build_dep(path) do
    name = Path.basename(path)

    {
      String.to_atom(name),
      path: path,
    }
  end

  defp aliases do
    [
      deps_all: [
        "deps.get",
        &mk_deps/1
      ]
    ]
  end
end

NOTE! We are still required to call mix deps_all (or mix deps.get) before mix deps.compile.

Conclusion

I think it’s a neat feature in mix that multiple build tools can be used and even though the proposed solutions are “hacks” it does get the job done.

Introduction¶

The problem¶

Conclusion¶

Introduction

The problem

Conclusion