A ZeroMQ Protocol for Remctl

Here are some initial notes on writing a ZeroMQ backend for remctl. This is more of a quick brain-dump of ideas. The idea behind this is that often I use remctl as an authenticated communication layer, but it's native method of "fork a program, return the output" is painful for things that have expensive setup times (establishing a database connection, etc.), so I often end up writing a shim that talks down a socket to a persistent daemon. Having done too much local socket programming, I'd like to use ZeroMQ to make life much easier.

  1. Modify remctld to support talking to a ZeroMQ req/rep socket using the protocol below.
  2. Add an option to the remctld configuration, "type=", with values "exec" (the default) and "zeromq". If the type is "exec", the "executable" is just that. If it is "zeromq", the "executable" is treated as a ZeroMQ req/rep socket talking the protocol below.

The protocol:

  1. Send a multipart ZeroMQ request with the following components:
    1. The string "ARGUMENT", a string representation of the number of arguments to follow in the next part, and then each argument in subsequent parts. As is the convention in remctl, the subcommand is the first argument.
    2. The string "ENVIRONMENT", with a string representation of the number of environment variables to follow in the next part, then, for each environment variable, the name and value in separate parts.
  2. Return the following components, again in a ZeroMQ multipart message:
    1. For each chunk of output, return three parts:
      1. The string "OUTPUT"
      2. A string reprentation of the output stream number. Note: while the remctl protocol supports many output streams, the stock remctl client only supports STDOUT (1) and STDERR (2)).
      3. The output
    2. Finally, return either "RETURNCODE" and in the following part a string representation of the return code, or "ERRORCODE" and in the following part a string representation of a remctl MESSAGE_ERROR error code.

Example exchange: note that the parts of the ZeroMQ message are separated by |.

ARGUMENT|3|foo|bar|baz|ENVIRONMENT|4|REMOTE_USER|kula@TPROA.NET|REMOTE_ADDR|204.11.35.93|REMOTE_HOST|keymaster.tproa.net|REMCTL_COMMAND|test

OUTPUT|1|Here is some sample output\n|OUTPUT|2|Error output\n|OUTPUT|2|More err\n|OUTPUT|1|Another line\n|RETURNCODE|23

This is equivalent to a user kula@TPROA.NET on host keymaster.tproa.net with address 204.11.35.93 running remctl [remserver] test foo bar baz and getting back "Here is some sample output\n" on STDOUT, "Error output\n" and "More err\n" on STDERR, the line "Another line\n" on STDOUT and exiting with a return code of 23.

Random notes:

  1. Because of ZeroMQ's all-or-nothing method of operation, there won't be any stream of output back to the client. It will get all bundled up and sent back in one chunk via remctld.
  2. We count arguments and environment variables because we know them, but we don't count output to simplify handlers — they can just keep gloming on output until they are done.
  3. I represent all the integers above as strings so you don't have to worry about which byte-ordering integers are in, or how many bits to expect.
  4. The "ERRORCODE" option is primarily there so you can return ERROR_ACCESS from your backend, based on the user/host/addr making the connection. Or ERROR_UNKNOWN_COMMAND, I guess.
  5. To make the message smaller, I'm willing to accept the following string substitutions:
    1. ARGUMENT -> A
    2. ENVIRONMENT -> E
    3. OUTPUT -> O
    4. RETURNCODE -> R
    5. ERRORCODE -> E