Development and testing with Docker¶

Arvados is primarily intended as a system to be run on large server clusters. For development and testing purposes, we have provided a process to build and deploy the Arvados system in Docker containers on a single machine.

http://doc.arvados.org/install/install-docker.html shows how to install and update a development system.

Additional sources of information¶

The docker/README.md file in the source distribution has more notes.

Installing the client libraries¶

./install_sdk.sh (in the docker directory) will install the CLI tools like arv and arv-put. If that does not work for you, try installing these by hand:

• RVM -- if you do not already have RVM installed on your workstation, install it now.
  

  sudo apt-get install curl libcurl3 libcurl3-gnutls libcurl4-openssl-dev python-pip
  curl -sSL https://get.rvm.io | bash -s stable
  source ~/.rvm/scripts/rvm
  rvm install 2.1.0
  

• The arvados-cli gem:
  

  gem install arvados-cli
  sudo pip install --upgrade httplib2
  

Debugging containers¶

Once it's running, you can get run a shell inside a container using nsenter.

Install nsenter:

docker run --rm jpetazzo/nsenter cat /nsenter > /tmp/nsenter
sudo install -m 0755 /tmp/nsenter /usr/local/bin/

Install this shortcut:

sudo tee /usr/local/bin/nse <<'EOF'
#!/bin/sh
PID=$(docker inspect --format {{.State.Pid}} "$*")
exec nsenter --target $PID --mount --uts --ipc --net --pid
EOF

chmod +x /usr/local/bin/nse

Then you can run a shell in a container like this:

nse keep_server_0

Console¶

./arvdock starts containers in detached mode. You can attach your terminal to them in order to see stdout/stderr:

docker attach --sig-proxy=false keep_server_0

Press control-C to detach. (If you didn't use --sig-proxy=false, control-C will send SIGINT into the container.)

Tutorials¶

Once you have installed the servers and started them with arvdock start, and installed the client libraries, you should be able to work through the user tutorials (using your host instead of a shell VM) to upload data into Arvados and run Crunch jobs in local development mode.