tunnel

Introduction

Erlang OTP includes a SSH application I thought I would attempt to implement SSH tunneling in Elixir, similar to how we can define tunnels using openSSH.

With openSSH one can define a local tunnel using the following command:

ssh -nNT -L 8585:127.0.0.1:9000 user@192.168.90.1

This will forward any traffic on localhost:8585 to 127.0.0.1:9000 through 192.168.90.1. In other words localhost:8585 will connect to 192.168.90.1:9000.

  • The -nNT option tells SSH to run no command, redirect Null to stdin, and not allocate a TTY. So it’s not possible to run any commands through the tunnel.
  • -L is for Local bind of the address,
  • -R is the same as -L but in reverse, it even works for sockets too: ssh -nNT -L 9000:/var/lib/mysql/mysql.sock user@192.168.90.1

TLDR Full source can be found here.

2021 Update This article was writen in 2018 ssh tunnels are now supported in OTP (from OTP 22 I think?) using tcpip_tunnel_from_server and tcpip_tunnel_to_server.

Let’s get started!

Connecting to a ssh server

We’ll define a module with a simple connect to create a connection to a ssh server:

# lib/ssht.ex
defmodule SSHt do
  def connect(opts \\ []) do
    host = Keyword.get(opts, :host, "127.0.0.1")
    port = Keyword.get(opts, :port, 22)
    ssh_config = defaults(opts)

    :ssh.connect(String.to_charlist(host), port, ssh_config)
  end

  defp defaults(opts) do
    user = Keyword.get(opts, :user, "")
    password = Keyword.get(opts, :password, "")

    [
      user_interaction: false,
      silently_accept_hosts: true,
      user: String.to_charlist(user),
      password: String.to_charlist(user)
    ]
  end
end

Since we are calling the into erlang the host string (binary) needs to be encoded as a charlist. Here we are using username / password for authentication as well as not allowing user interactions, any options requiring interactions will fail the connection attempt. There are a lot more options available for more fine grained control, but for us these options will suffice.

The ssh application needs to be started before we try to connect to the server:

# mix.exs
def application do
  [
    extra_applications: [:logger, :ssh]
  ]
end

And we can try it out:

iex(1)> {:ok, pid} = SSHt.connect(host: "192.168.90.15", user: "ubuntu", password: "")

{:ok, #PID<0.181.0>}

It works! With this we are ready to start implementing the tunnels!

Types of tunnels

There are two kinds of tunnels which we are interested in:

directtcp-ip allows us (the client) to connect to an ip:port using the ssh server and direct-streamlocal allows us to connect to unix domain socket. directtcp-ip forwarding has been a part of the ssh application in the past but has since been removed, however we can implement it by using :ssh_connection_handler:open_channel/6

This is the function used internally for creating channels (:ssh_connection.session_channel/2/4 for instance).

For reference this is what it looks like:

:ssh_connection:open_channel(ConnectionHandler, ChannelType, ChannelSpecificData, InitialWindowSize, MaxPacketSize, Timeout)

  • ConnectionHandler is the pid we receive from :ssh.connect
  • ChannelType is the type of message for us this will be either “dirrect-tcpip” or “direct-streamlocal@openssh.com”
  • ChannelSpecificData is the messag we’ll construct from the message format below
  • InitalWindowSize Initial TCP window size
  • MaxPacketSize Max allowed packet size

and the directtcp-ip message format:

byte      SSH_MSG_CHANNEL_OPEN
string    "direct-tcpip"
uint32    sender channel
uint32    initial window size
uint32    maximum packet size
string    host to connect
uint32    port to connect
string    originator IP address
uint32    originator port

We’ll define a direct_tcpip/3 function in ssht.ex

# lib/ssht.ex
defmodule SSHt do
  @ini_window_size 1024 * 1024
  @max_packet_size 32 * 1024
  @direct_tcpip String.to_charlist("direct-tcpip")

  def connect(opts \\ []) do
    host = Keyword.get(opts, :host, "127.0.0.1")
    port = Keyword.get(opts, :port, 22)
    ssh_config = defaults(opts)

    :ssh.connect(String.to_charlist(host), port, ssh_config)
  end

  def direct_tcpip(conn, from, to) do
    {orig_host, orig_port} = from
    {remote_host, remote_port} = to

    remote_len = byte_size(remote_host)
    orig_len = byte_size(orig_host)

    msg = <<
      remote_len::size(32),
      remote_host::binary,
      remote_port::size(32),
      orig_len::size(32),
      orig_host::binary,
      orig_port::size(32)
    >>

    :ssh_connection_handler.open_channel(
      conn,
      @direct_tcpip,
      msg,
      @ini_window_size,
      @max_packet_size,
      :infinity
    )
  end

  defp defaults(opts) do
    user = Keyword.get(opts, :user, "")
    password = Keyword.get(opts, :password, "")

    [
      user_interaction: false,
      silently_accept_hosts: true,
      user: String.to_charlist(user),
      password: String.to_charlist(password)
    ]
  end
end

On line 21 we create a message by translating from directtcp-ip message format to a binary, due to the excellent bit syntax it reads basically the same as the original message format from the specification. Since the host fields can be of variable size, the length is prepended. Note that the SSH_MSG_CHANNEL_OPEN and sender channel fields are not part of our message. These will be set internally in :ssh_connection_handler.open_channel.

If you’re interested in reading more about binary pattern matching I think this article does a good job explaining it.

The type @direct_tcpip is defined as a module attribute, remember since we are calling an erlang application it needs to be represented as a charlist instead of a string. @max_window_size is set to the 32k as specified in rfc4253

6.1. Maximum Packet Length All implementations MUST be able to process packets with an uncompressed payload length of 32768 bytes or less and a total packet size of 35000 bytes or less (including ‘packet_length’, ‘padding_length’, ‘payload’, ‘random padding’, and ‘mac’). ….

@ini_window_size is trickier since I don’t know the impact of setting a value which is too low (or too high). We’ll set it to 105kb since I’m pretty sure I’ve seen it set to 1024 * 1024 somewhere so we just go with that. If our call to :ssh_connection_handler.open_channel/6 is successful we’ll receive a {:open, channel}. We’ll use the channel and the connection pid to call :ssh_connection.send/3 and send data to our forwarded ip. Let’s try it out by sending a raw HTTP message:

iex(1)>  {:ok, pid} = SSHt.connect(host: "192.168.90.15", user: "ubuntu", password: "")
{:ok, #PID<0.166.0>}
iex(2)> data = "GET / HTTP/1.1\r\nHost: localhost:8080\r\nUser-Agent: ssht/0.0.1\r\nAccept: */*\r\n\r\n"
iex(3)>  {:open, ch} = SSHt.direct_tcpip(pid, {"127.0.0.1", 8080}, {"192.168.90.15", 80})
{:open, 0}
iex(4)> :ssh_connection.send(pid, ch, data)
# handle the data returned on the connection with a receive block
iex(4)> receive do
...(4)> {:ssh_cm, _, {:data, ^ch, _, data}} -> IO.puts("#{data}")
...(4)> end
end

We can receive messages using a receive block or by creating the channel inside a process (GenServer for instance) and receive it using handle_info/2 callback:

def handle_info({:ssh_cm, _, {:data, _ch, _, data}}, state) do
  IO.puts("Received data #{length(data)}"}
  {:noreply, state}
end

NOTE! You need to have something actually responding on the forwarded ip otherwise it will fail. I have a VM setup with a private ip 192.168.90.15 with nginx running on port 80.

So far we’ve achieved:

  • Connecting to a ssh server
  • Creating a forwarded directtcp-ip channel which we can read & write to

Next we’ll create a TCP server to relay traffic from our host to the ssh server.

A TCP Server

For this I’m using ranch, add ranch to the deps:

# mix.exs
defp deps do
  [
    {:ranch, "~> 1.4"}
  ]
end

For this part we are going to do the following:

  • On demand TCP-servers
  • Relay traffic from a TCP client to forwarded ssh channel and back
  • Allow connecting using ip:port and a unix domain socket

The supervisor

The TCP servers need to be started on demand for this we’ll use DynamicSupervisor. To quote the docs

A DynamicSupervisor starts with no children. Instead, children are started on demand via start_child/2

Create an application module at lib/ssht/application.ex and put the following in it:

# lib/ssht/application.ex
defmodule SSHt.Application do
  @moduledoc """
  Application module
  """
  use Application

  def start(_type, _args) do
    children = [
      {DynamicSupervisor, name: SSHt.TunnelSupervisor, strategy: :one_for_one}
    ]

    Supervisor.start_link(children, strategy: :one_for_one)
  end
end

Child processes can now be started using start_child/2 like: DynamicSupervisor.start_child(SSHt.TunnelSupervisor, {MyGenServer, []})

First argument is the supervisor name, second is a child_spec. As long as the module implements a child_spec/1 one can use the tuple shorthand. GenServer, Task, Supervisor all implement this so if you’re deriving your module from any of those you’re good to go. The second argument is the arguments that will be passed to the child being started. In the example above we have no arguments so we specify an empty list.

We’ll define lib/ssht/tunnel.ex to use as our interface for creating TCP listeners.

# lib/ssht/tunnel.ex
defmodule SSHt.Tunnel do
  @type to :: {:tcpip, tuple()} | {:local, String.t}

  @spec start_link(pid(), to) :: {:ok, pid()} | {:error, term()}
  def start_link(ref, to) do
    DynamicSupervisor.start_child(
      SSHt.TunnelSupervisor,
      worker_spec(worker_opts(ref, to))
    )
  end

  defp worker_spec(opts) do
    name = Keyword.get(opts, :name)

    ranch_opts =
      case Keyword.get(opts, :target) do
        {:local, path} -> [{:local, path}]
        {:tcpip, {port, _}} -> [{:port, port}]
      end

    :ranch.child_spec(
      name,
      100,
      :ranch_tcp,
      ranch_opts,
      SSHt.Tunnel.TCPHandler,
      opts
    )
  end

  defp worker_opts(ref, {:tcpip, {port, _}} = to), do: basic_opts(ref, base_name(port), to)

  defp worker_opts(ref, {:local, socket_path} = to),
    do: basic_opts(ref, base_name(socket_path), to)

  defp basic_opts(ref, name, target) do
    Keyword.new()
    |> Keyword.put(:name, name)
    |> Keyword.put(:ssh_ref, ref)
    |> Keyword.put(:target, target)
  end

  defp base_name(port_or_path) do
    "#{__MODULE__}.#{port_or_path}" |> String.to_atom()
  end
end

start_link/2 accepts a ssh connection pid and a tuple which can either be {:tcpip, {local_port, {ip_addr, remote_port}} or {:local, socket_path}.

We use :ranch.child_spec in worker_spec/2 to create the specification for our TCP listener. worker_spec takes name, target from the options list and pattern matches on the target to determine the kind of listener we should use, :tcpip for starting a port listener and :local for a domain socket listener.

We have yet to define the SSHt.Tunnel.TCPHandler which will be the module responsible for handling a TCP client connection. So let’s do that:

# lib/ssht/tunnel/tcp_handler.ex
defmodule SSHt.Tunnel.TCPHandler do
  use GenServer

  def start_link(ref, socket, transport, opts) do
    pid = :proc_lib.spawn_link(__MODULE__, :init, [{ref, socket, transport, opts}])
    {:ok, pid}
  end

  def init({ref, socket, transport, opts}) do
    target = Keyword.get(opts, :target)
    ssh_ref = Keyword.get(opts, :ssh_ref)

    {:open, channel} = ssh_forward(ssh_ref, target)
    :ok = :ranch.accept_ack(ref)
    :ok = transport.setopts(socket, [{:active, true}])

    :gen_server.enter_loop(__MODULE__, [], %{
      socket: socket,
      transport: transport,
      ssh_ref: ssh_ref,
      channel: channel
    })
  end

  defp ssh_forward(ref, target) do
    case target do
      {:local, path} -> SSHt.stream_local_forward(ref, path)
      {:tcpip, {port, to}} -> SSHt.direct_tcpip(ref, {"127.0.0.1", port}, to)
    end
  end
end

As you’ve probably already seen this looks a little bit different than usually. The reason for using :proc_lib.spawn_link is due to how GenServer.start_link works. start_link does not return until the init returns. Calling :ranch.accept_ack would cause a deadlock. We use :gen_server.enter_loop/3 to fallback to the normal GenServer execution loop after the initialization. (More here)

Time to receive TCP messages and send them onto the forwarded channel:

# lib/ssht/tunnel/tcp_handler.ex
defmodule SSHt.Tunnel.TCPHandler do
  use GenServer
  require Logger

  def start_link(ref, socket, transport, opts) do
    pid = :proc_lib.spawn_link(__MODULE__, :init, [{ref, socket, transport, opts}])
    {:ok, pid}
  end

  def init({ref, socket, transport, opts}) do
    clientname = stringify_clientname(socket)
    target = Keyword.get(opts, :target)
    ssh_ref = Keyword.get(opts, :ssh_ref)

    {:open, channel} = ssh_forward(ssh_ref, target)
    :ok = :ranch.accept_ack(ref)
    :ok = transport.setopts(socket, [{:active, true}])

    :gen_server.enter_loop(__MODULE__, [], %{
      socket: socket,
      transport: transport,
      ssh_ref: ssh_ref,
      channel: channel,
      clientname: clientname
    })
  end

  def handle_info(
        {:tcp, _, data},
        %{ssh_ref: ssh, channel: channel, clientname: clientname} = state
      ) do
    :ok = :ssh_connection.send(ssh, channel, data)
    Logger.info(fn -> "Message from: #{clientname}: #{inspect(data)}." end)

    {:noreply, state}
  end

  def handle_info({:tcp_error, _, reason}, %{clientname: clientname} = state) do
    Logger.info(fn -> "Error #{clientname}: #{inspect(reason)}" end)
    {:stop, :normal, state}
  end

  def handle_info(
        {:tcp_closed, _},
        %{clientname: clientname, ssh_ref: ssh, channel: channel} = state
      ) do
    Logger.info(fn -> "Client #{clientname} disconnected channel #{channel}" end)

    {:stop, :normal, state}
  end

  def handle_info(
        {:ssh_cm, _, {:data, _, _, data}},
        %{socket: socket, transport: transport} = state
      ) do
    :ok = transport.send(socket, data)
    {:noreply, state}
  end

  def handle_info({:ssh_cm, _, {:eof, _channel_id}}, state) do
    {:stop, :normal, state}
  end

  def terminate(reason, %{ssh_ref: ssh, channel: channel}) do
    :ok = :ssh_connection.close(ssh, channel)
    Logger.info("terminated reason #{inspect(reason)}")
  end

  defp ssh_forward(ref, target) do
    case target do
      {:local, path} -> SSHt.stream_local_forward(ref, path)
      {:tcpip, {port, to}} -> SSHt.direct_tcpip(ref, {"127.0.0.1", port}, to)
    end
  end

  defp stringify_clientname(socket) do
    {:ok, {addr, port}} = :inet.peername(socket)

    address =
      addr
      |> :inet_parse.ntoa()
      |> to_string()

    "#{address}:#{port}"
  end
end

Whenever we get a tcp message we pattern match on {:tcp,_, data} and send data on to the ssh channel, when we receive a {:ssh_cm, _ {:data,_,_, data}} message we send it on to the tcp socket. We also implement the terminate/2 callback to close the ssh channel when the client closes the socket, a EOF is received or for any unexpected behavior.

There’s a potential problem with this implementation however, since we need to differentiate between the connections there’s a channel created for every TCP connection. I’m not certain if it is a problem our not, initially I thought I would define a tunnel (channel) per TCP server, but as the channel is linked to the process creating it the TCP server would receive all ssh messages with no apparent way of telling messages apart. By doing it this way, we’re certain that the ssh messages received are destined to the correct TCP client.

Conclusion / Final words

I think it’s pretty cool that we are able to implement this in pure Elixir in about 200 lines. It’s just a testament to how incredible Erlang/OTP and Elixir really is!

The source in it’s entirety can be found here.