Arvados

h1. Vault

Third proposal revision on how to use Vault from an Arvados container, based on additional discussion and research:

# User creates a "secret" object in Arvados (see "Secrets table" below)

# User logs into Vault using their Arvados API token

# User reads/writes password in Vault at "path" designated by the "secret" object.

# Container, when run, logs into Vault using its container run token

# Vault uses Arvados plugin to associate container token with an Arvados user and determine policies

# Container looks up entries in the "secrets" table to get the path

# Container queries vault for the credentials it needs

# Vault allows/denies query based on polices previously returned by Arvados plugin

Notes:

* Credentials may be passed in via environment, config file or command line like any other input, so it would be nice to generalize this as a CWL feature where some inputs are simply marked "sensitive" and automatically handled by Vault.  However, while arvados-cwl-runner is responsible for constructing the container request it doesn't run on the compute node, only crunch-run, which is "dumb".  One idea is for crunch-run to perform string substitution of symbol with secret the environment, command line, or config file on the compute node.

h2. Vault plugins

We can implement Arvados authentication as a plugin.  A plugin is a service which is executed by Vault and communicates with Vault over RPC.  Vault provides infrastructure to do this easily in Go.

https://www.hashicorp.com/blog/building-a-vault-secure-plugin

Authentication returns a list of Vault policies to apply to the resulting login.  This means policies need to be stored in Arvados (see above discussion).

If the plugin can return arbitrary policies, this suggests that Arvados theoretically has unlimited ability to grant access to the Vault database.  We should consider how to restrict it to just those secrets / policies intended to be managed by Arvados.

h2. Secrets table

If we add a new "secrets" table we can manage access using our existing permission model.  This means it is managed by the owner, and we can create read/write permission links.  The secrets table would consist only of the standard arvados fields: uuid, owner_uuid, and also a "name" and read-only "path" field.  Its purpose is just to be a node in the permission graph.

The Arvados vault plugin would query Arvados using the token it receives and get back a list of secrets the token is allowed to see.  The secrets are stored in Vault under the Arvados uuid for each "secret" object.

To look up a secret, looks it up by owner_uuid + name or uuid to get the record, which gives the Vault "path" and then asks for the secret from vault.

Ideally, token scopes could also be used to restrict the "secrets" listing to only returns specific credentials (however, I don't think Arvados scopes can currently do this).  In any event, we don't want a scoped token to have access to "secrets" unless explicitly given.

h2. Defining policies

The Arvados Vault plugin will need to define policies on the fly.  Initial thought is to create a policy for each secret, eg "can_read_xyz" and the user policy list consists of one policy per secret the user is allowed to access.  (This may present a scaling issue if the number of secrets is large).  The authentication plugin could do this whenever it sees a "secret" uuid that doesn't have a corresponding policy (this seems preferrable to having the API server do it, or a cron job, or ...)

Note: we don't want to give permission to write arbitrary policies, which would allow access to operational secrets (multi-tenancy).  Can we define a policy which applied to the plugin itself, which only allows it to create policies for Arvados and not access the rest of the system?