Arvados

h1. Vault

Second pass at proposal on how to use Vault from an Arvados container, based on Feb 13 meeting:

* Credentials are entered into Vault

* Arvados user is granted permission to read/write certain credentials.  Need to determine how this works.

** Who can grant it?

** How is it represented?

** Some ideas: associated with a specific token (like "scopes"); create a new "secrets" table and use permission links

** Needs to be carefully engineered so that users can't manipulate policies to claim secrets they shouldn't have access to.

* Container, when run, logs into Vault using its container run token

* Vault uses Arvados plugin to associate container token with an Arvados user and determine policies

* Container queries vault for the credentials it needs

* Vault allows/denies query based on polices previously returned by Arvados plugin

Notes:

* Credentials may be passed in via environment, config file or command line like any other input, so it would be nice to generalize this as a CWL feature where some inputs are simply marked "sensitive" and automatically handled by Vault.  However, while arvados-cwl-runner is responsible for constructing the container request it doesn't run on the compute node, only crunch-run, which is "dumb", so its not clear how we go about allowing for arbitrary substitution into the environment, command line, or config file on the compute node.

* Response wrapping is another mechanism provided by Vault for delegating access.  A wrapped response can be passed through and accessed exactly once (would imply max_container_count: 1).

* Another solution is to develop an AuthMethod for vault with Arvados Token proivded by the container. 

h2. Vault plugins

We can implement Arvados authentication as a plugin.  A plugin is a service which is executed by Vault and communicates with Vault over RPC.  Vault provides infrastructure to do this easily in Go.

https://www.hashicorp.com/blog/building-a-vault-secure-plugin

Authentication returns a list of Vault policies to apply to the resulting login.  This means policies need to be stored in Arvados (see above discussion).

If the plugin can return arbitrary policies, this suggests that Arvados theoretically has unlimited ability to grant access to the Vault database.  We should consider how to restrict it to just those secrets / policies intended to be managed by Arvados.

h2. Secrets table

If we add a new "secrets" table we can manage access using our existing permission model.  This means it is managed by the owner, and we can create read/write permission links.  The secrets table would consist of 

* standard arvados fields: uuid, owner_uuid, etc

* the path to the secret in Vault

The Arvados vault plugin would query Arvados using the token it receives and get back a list of secrets the token is allowed to see.  Ideally we could use token scopes to restrict listing to only returns specific credentials (however, I don't think scopes can currently do this)

h2. Defining policies

The Arvados Vault plugin will need to define policies on the fly.  Initial thought is to create a policy for each secret, eg "can_read_xyz" and the user policy list consists of one policy per secret the user is allowed to access.  (This may present a scaling issue if the number of secrets is large).

Note: we don't want to give permission to write arbitrary policies, which would allow access to operational secrets (multi-tenancy).  Can we define a policy which applied to the plugin itself, which only allows it to create policies for Arvados and not access the rest of the system?