Using Keep with Azure Storage (BETA)¶

Starting at #7241 keepstore can use Azure Storage containers as storage devices. Each data block is stored as a "Block Blob".

Features:

• You can configure multiple Azure volumes
• You can mix Azure and POSIX volumes
• It is safe to share Azure volumes between multiple keepstore processes/hosts
• Azure volumes can be marked readonly

Missing features:

• The -serialize flag is not supported
• There is no way to control the way the data is organized (named) in the container. The name of each blob is the hash of the corresponding data, e.g., 73feffa4b7f6bb68e44cf984c85f6e88.

Setup¶

Outline:

1. Install the azure CLI tool.
2. Set credentials.
   azure login
3. Set cli tool mode.
   azure config mode arm
4. Create a resource group (unless you're going to use an existing one, of course).
   azure group create examplegroupname eastus
5. Create a storage account (ditto). The --type argument determines storage replication policy; see docs
   azure storage account create --type LRS --location eastus --resource-group examplegroupname exampleaccountname
6. Get storage account keys:
   azure storage account keys list --resource-group examplegroupname exampleaccountname
   (This will give you a base64-encoded key looking something like t3wfMAZ4/YBso7Jr5dtaR7gdrSJmdqzIv1iLofr/2xkZLqLwjj3iwV1YNYbjPUhewXYpp6KxmJUH9L3cfLALtw==)
7. Create a container:
   AZURE_STORAGE_ACCOUNT=exampleaccountname \
AZURE_STORAGE_ACCESS_KEY="t3wfMAZ4/YBso7Jr5dtaR7gdrSJmdqzIv1iLofr/2xkZLqLwjj3iwV1YNYbjPUhewXYpp6KxmJUH9L3cfLALtw==" \
azure storage container create examplecontainername

Configuring keepstore¶

Store the account key in a file with suitable permissions. (A trailing newline will be ignored, but don't put any other characters in there.)

cd /etc/sv/keepstore
(umask 077; vi exampleaccountname.key)

Update your run script. If you are already running with some local volumes, your run script might have this:

keepstore \
 -volume /data/disk0 -volume /data/disk1

If you want to change your local volumes to be readonly, and use the azure container to write new data, you'd change it to this:

keepstore \
 -readonly \
 -volume /data/disk0 -volume /data/disk1 \
 -readonly=false \
 -azure-storage-account-key-file ./exampleaccountname.key \
 -azure-storage-account-name exampleaccountname \
 -azure-storage-container-volume examplecontainername

If you have multiple containers and some of them are in different accounts, you might have something like this:

keepstore \
 -readonly \
 -volume /data/disk0 -volume /data/disk1 \
 -readonly=false \
 -azure-storage-account-key-file ./firstaccount.key \
 -azure-storage-account-name firstaccount \
 -azure-storage-container-volume containerA \
 -azure-storage-container-volume containerB \
 -azure-storage-container-volume containerC \
 -azure-storage-account-key-file ./secondaccount.key \
 -azure-storage-account-name secondaccount \
 -azure-storage-container-volume containerX \
 -azure-storage-container-volume containerY \
 -azure-storage-container-volume containerZ

When invoked this way, keepstore uses the "firstaccount" credentials to connect to containerA/B/C and use the "secondaccount' credentials to connect to containerX/Y/Z.