• Table of contents
• Hacking prerequisites
  • Host options
    • Starting on your workstation
    • Starting on a VM
    • Starting in a docker container
  • Install dev environment
  • Get the arvados source tree and test scripts
  • Start Postgres
  • Start X11
  • Setup groups
  • Create a Postgres user
  • Run tests

Hacking prerequisites¶

The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.

Host options¶

Starting on your workstation¶

If your workstation is a debian jessie system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".

Other linux distributions should work too with some modifications, but it's probably easier to use a VM.

Starting on a VM¶

Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian jessie on it. The instructions below assume you have just a few basic requirements:

• SSH server
• sudo (apt-get install sudo)
• A user account with sudo privileges

Starting in a docker container¶

This can get you started quickly, but (unlike the above options) you'll need to remember to use something like docker commit to save your state before shutting down your container.

See http://docker.io for more about installing docker. On debian it looks something like this.

echo 'deb http://http.debian.net/debian jessie-backports main' | sudo tee -a /etc/apt/sources.list.d/jessie-backports.list
sudo apt-get install docker.io
sudo adduser $USER docker
# {log out & log back in}
groups
# {should include "docker"}

Start up a new container with jessie, make a new user and log in as that user:

docker run -it --privileged debian:jessie bash
apt-get update
apt-get -y install sudo
adduser me
adduser me sudo
sudo -u me -i

The "--privileged" is required in order for /dev/fuse to be accessible (without it, no tests that require FUSE will work).

Install dev environment¶

# only on debian 9 (stretch), to permit ruby 2.3 compilation (see https://github.com/rbenv/ruby-build/wiki#openssl-usrincludeopensslasn1_mach102-error-error-this-file-is-obsolete-please-update-your-software):
sudo apt-get install --no-install-recommends libssl1.0-dev

sudo apt-get install --no-install-recommends \
    bison build-essential cadaver fuse gettext git gitolite3 graphviz \
    iceweasel libattr1-dev libfuse-dev libcrypt-ssleay-perl libjson-perl \
    libcrypt-ssleay-perl libcurl3 libcurl3-gnutls libcurl4-openssl-dev \
    libjson-perl libpcre3-dev libpq-dev libpython2.7-dev libreadline-dev \
    libssl-dev libxslt1.1 libwww-perl linkchecker lsof nginx perl-modules \
    postgresql python python-epydoc pkg-config sudo virtualenv \
    wget xvfb zlib1g-dev libgnutls28-dev python3-dev \
    r-base r-cran-testthat libxml2-dev pandoc

# ruby 2.3
(
 set -e
 mkdir -p ~/src
 cd ~/src
 wget http://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.6.tar.gz
 tar xzf ruby-2.3.6.tar.gz
 cd ruby-2.3.6
 ./configure --disable-install-doc
 make
 sudo make install
 sudo gem install bundler
)

# go >= 1.9
sudo apt-get install golang-1.9 || \
(
 set -e
 wget https://storage.googleapis.com/golang/go1.9.4.linux-amd64.tar.gz
 sudo tar -C /usr/local -xzf go1.9.4.linux-amd64.tar.gz
 cd /usr/local/bin
 sudo ln -s ../go/bin/* .
)

# phantomjs 1.9.8
(
 set -e
 PJS=phantomjs-1.9.8-linux-x86_64
 wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
 sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
 sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
)

1. npm
   (
   set e
   wget -O https://nodejs.org/dist/v6.11.2/node-v6.11.2-linux-x64.tar.xz | sudo tar -C /usr/local -xJf -
   sudo ln -s ../node-v6.11.2-linux-x64/bin/{node,npm} /usr/local/bin/
   )

Note: For ubuntu, virtualenv is python-virtualenv

Get the arvados source tree and test scripts¶

cd
git clone https://github.com/curoverse/arvados.git

...or, if you're a committer with your public key on our git server:

cd
git clone git@git.curoverse.com:arvados.git

Start Postgres¶

If you're running in a docker container you'll need to start Postgres manually:

sudo /etc/init.d/postgresql start

(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)

Start X11¶

In order for the apps/workbench tests to function, firefox needs to have an X11 server (or it will fail to start). If you are on a workstation with a "real" X server, that should work. If not, Xvfb is an X server that renders to a virtual framebuffer so that selenium/firefox tests can run in headless mode. Also make sure that the DISPLAY environment variable is set accordingly.

Xvfb :0.0 &
export DISPLAY=":0.0" 

Setup groups¶

Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them.

Create a Postgres user¶

Create an "arvados" user with "create database" privileges. The test suite will create and drop the arvados_test database as needed.

newpw=`tr -cd a-zA-Z </dev/urandom |head -c32`
sudo -u postgres psql -c "create user arvados with createdb encrypted password '$newpw'" 
cp -i ~/arvados/services/api/config/database.yml{.example,}
newpw="$newpw" perl -pi~ -e 's/xxxxxxxx/$ENV{newpw}/' ~/arvados/services/api/config/database.yml

Run tests¶

time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados

During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with --skip-install or --only-install sdk/ruby and so on.

mkdir -p ~/.cache/arvados-build
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build