Arvados

h1. Using Keep with Azure Storage (BETA)

Starting at #7241 keepstore can use Azure Storage containers as storage devices. Each data block is stored as a "Block Blob".

Features:

* You can configure multiple Azure volumes

* You can mix Azure and POSIX volumes

* It is safe to share Azure volumes between multiple keepstore processes/hosts

* Azure volumes can be marked readonly

Missing features:

* The @-serialize@ flag is not supported

* There is no way to control the way the data is organized (named) in the container. The name of each blob is the hash of the corresponding data, e.g., @73feffa4b7f6bb68e44cf984c85f6e88@.

h2. Setup

Outline:

# Install the azure CLI tool.

# Set credentials.

@azure login@

# Set cli tool mode.

@azure config mode arm@

# Create a resource group (unless you're going to use an existing one, of course).

@azure group create examplegroupname eastus@

# Create a storage account (ditto). The @--type@ argument determines storage replication policy; see "docs":https://azure.microsoft.com/en-us/documentation/articles/storage-introduction/#replication-for-durability-and-high-availability

@azure storage account create --type LRS --location eastus --resource-group examplegroupname exampleaccountname@

# Get storage account keys:

@azure storage account keys list --resource-group examplegroupname exampleaccountname@

(This will give you a base64-encoded key looking something like @t3wfMAZ4/YBso7Jr5dtaR7gdrSJmdqzIv1iLofr/2xkZLqLwjj3iwV1YNYbjPUhewXYpp6KxmJUH9L3cfLALtw==@)

# Create a container:

@AZURE_STORAGE_ACCOUNT=exampleaccountname \@

@AZURE_STORAGE_ACCESS_KEY="t3wfMAZ4/YBso7Jr5dtaR7gdrSJmdqzIv1iLofr/2xkZLqLwjj3iwV1YNYbjPUhewXYpp6KxmJUH9L3cfLALtw==" \@

@azure storage container create examplecontainername@

h2. Configuring keepstore

Store the account key in a file with suitable permissions. (A trailing newline will be ignored, but don't put any other characters in there.)

<pre>

cd /etc/sv/keepstore

(umask 077; vi exampleaccountname.key)

</pre>

Update your run script. If you are already running with some local volumes, your run script might have this:

<pre>

keepstore \

 -volume /data/disk0 -volume /data/disk1

</pre>

If you want to change your local volumes to be readonly, and use the azure container to write new data, you'd change it to this:

<pre>

keepstore \

 -readonly \

 -volume /data/disk0 -volume /data/disk1 \

 -readonly=false \

 -azure-storage-account-key-file ./exampleaccountname.key \

 -azure-storage-account-name exampleaccountname \

 -azure-storage-container-volume examplecontainername

</pre>

If you have multiple containers and some of them are in different accounts, you might have something like this:

<pre>

keepstore \

 -readonly \

 -volume /data/disk0 -volume /data/disk1 \

 -readonly=false \

 -azure-storage-account-key-file ./firstaccount.key \

 -azure-storage-account-name firstaccount \

 -azure-storage-container-volume containerA \

 -azure-storage-container-volume containerB \

 -azure-storage-container-volume containerC \

 -azure-storage-account-key-file ./secondaccount.key \

 -azure-storage-account-name secondaccount \

 -azure-storage-container-volume containerX \

 -azure-storage-container-volume containerY \

 -azure-storage-container-volume containerZ

</pre>

When invoked this way, keepstore uses the "firstaccount" credentials to connect to containerA/B/C and use the "secondaccount' credentials to connect to containerX/Y/Z.