Credential storage¶

Background¶

As part of implementing a-c-r natively supports S3 inputs just like HTTP/S and Objects as pseudo-blocks in Keep, we would like a way to store credentials so that Arvados can authenticate to other systems, e.g. AWS S3.

The current system for managing secrets is specific to workflows and deletes the secret as soon as the workflow is finished. For this external data access feature, we require a more persistent credential storage system that can be accessed by keepstore.

User perspective:

Want to be able to manage credentials in workbench, and then Arvados services that need it can look it them up. The motivating use case is AWS credentials that have a key id/key secret pair (much like arvados API key uuid / secret) so that we can easily access objects in external S3 buckets.

Requirements¶

• Secrets should have an id for what type of thing they are, e.g. AWS credentials
• Secrets should have an optional scope. E.g. want to be able to provide different credentials for different resources, buckets, etc.
• Should the secret material itself be simple text column or a JSON object? For example AWS secret id/secret is a pair
• Different users should have different views of what secrets are available based on Arvados permissions. User should be able to share secrets at different levels of access, e.g.
  • can_read -- system services can fetch the credential on behalf of the user, but they cannot fetch it directly through the API
  • can_write -- user can update the credential, but still not read it back
  • can_manage -- user can grant permissions to the credential, but still not read it back
• Secrets should be write-only as much as possible, system services can retrieve secrets, but users cannot except in special circumstances
  • want a way to use secrets in workflows, which means they can be exposed if developers are careless. This is true of our current secrets support as well (it's inherently impossible to prevent it from being leaked in user-provided code if someone is really trying, but we'll at least be able to keep a record of which workflows accessed those secrets).

Security¶

Start with our threat model.

These are not passwords that we need to validate ourselves, these are credentials that will be provided to other services on behalf of the user, which means we have to be able to get them in the clear, we can't hash them. Unfortunately a google search for "how to store secrets in a database" comes up dozens of pages telling you not to store cleartext passwords and how to hash passwords and not so much advice on how to do what we need to do.

Ways credentials could leak

• Attacker uses Arvados API as a normal user
  • Should be restricted accessing credentials by normal access controls.
  • As previously noted, if we want to provide credentials to a user-supplied workflows, it is impossible defend against, so we have to exclude consider users who are authorized to use the credentials being able to do anything they want with those credentials from the threat model
• Attacker uses Arvados API as a superuser
  • Admins can already mostly access anything
  • The existing secret_mounts only makes it inconvenient for admins, if they can access the container's runtime token, they can fetch secret mounts
  • Boxing out admins via the the API is probably possible but may require sealing additional holes (e.g. placing stricter limits on admins accessing API tokens of other users)
• Attacker gains access to the database
  • Would be able to use SQL to read any column. E.g. currently secret_mounts is not encrypted, so it would be vulnerable.
  • To block this, columns need to be encrypted.
• Attacker gains access to the node the database is running on
  • Same as remote database access, except attacker additionally has access to the /etc/arvados/config.yml and any credentials kept in there.
• Attacker can intercept communications with the database and/or API server
  • This is probably game over for our entire security model, not just secrets handling. We rely on TLS to prevent this.

Analysis¶

I think we have to basically trust the integrity of the database and network. Storing them in the database, making them effectively write-only through the default API and only retrievable by containers adds some friction that makes it less likely to leak secrets without going too much into security theater. This is similar to the model that we are using with secret_mounts, and similar to the model used by Jenkins.

If we do decide we care about protecting secrets in database dumps or people using the psql console (but for some reason don't also have access to the decryption key), we could separately encrypt the database column. There is a pgcrypto module that provides functions that can be used directly in the query (not sure how that plays with Rails). This would require having the decryption key on hand in the API server or controller. I feel like if an attacker has access to a database dump or the psql console, we probably have bigger problems.

Implementation¶

New table "credentials" with the following columns:

field	type	description
name	string	Name assigned by the user, may be used for lookup
description	string	Optional description
credential_type	string	The type of credential being stored, for example "aws_access_key"
credential_id	string	The non-secret part of the credential, e.g. a username or AWS_ACCESS_KEY
credential_secret	string	The secret part of the credentials, e.g. a password or AWS_SECRET_ACCESS_KEY
expires_at	timestamp	Date at which the credential_secret field is automatically cleared, following this any attempts to access the credential_secret endpoint return an error.

For the first version, the owner_uuid for a credential must be the system root user. Credential names are unique for a given owner_uuid (which initially will only be the system root user).

In the future, it may be possible for credentials to be created and owned by other users.

Access is granted to credentials using inbound permission links. Normal Arvados permission semantics apply.

GET requests do not return credential_secret and the column cannot be used in list filters.

New API endpoint /arvados/v1/credentials/<uuid>/credential_secret permits fetching the secret, but only if the token is a container runtime token and the user has read access. Reading a secret like this should also emit an audit log.

Users will refer to credentials using either the uuid or the name. The client will look up the credential using the uuid or name (both of which will be unique in the first version of this feature, in future versions some kind of resolution may be necessary).

Using "group_class: credential" objects to indirectly grant access to external resources¶

Another area of future expansion.

I think this scheme can also be used to grant permission to external resources to which the Arvados system has access but needs to control user access. For example, instead of using AWS access key and secret, granting an AWS role.

In this case, we would do something like:

credential_type: aws_role
credential_id: the id of the role

There's no need for scope or secret here, but the Arvados service would check that the user has can_read on the credential to determine if it can grant the role to the user.

To prevent just anyone from creating aws_role objects, the service granting the AWS roles would only honor them if the aws_role credential is owned by the system user.