Project

General

Profile

Hacking prerequisites » History » Version 11

Joshua Randall, 09/11/2015 08:33 AM
adds requirement for an X11 server

1 1 Tom Clegg
{{>toc}}
2
3
h1. Hacking prerequisites
4
5
The Arvados test suite can run in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.
6
7
h2. Host options
8
9
h3. Starting on your workstation
10
11
If your workstation is a debian wheezy or jessie system -- and you don't mind installing a bunch of packages on your workstation, some of them without apt -- the easiest way to get running is to run tests on bare metal. Skip to "Dependencies".
12
13
Other linux distributions should work too with some modifications, but it's probably easier to use a VM.
14
15
h3. Starting on a VM
16
17
Another option is to create a virtual machine using something like Xen or VirtualBox, and run debian jessie on it. The instructions below assume you have just a few basic requirements:
18
* SSH server
19
* sudo (@apt-get install sudo@)
20
* A user account with sudo privileges
21
22
h3. Starting in a docker container
23
24
This can get you started quickly, but (unlike the above options) you'll need to remember to use something like @docker commit@ to save your state before shutting down your container.
25
26
See http://docker.io for more about installing docker. On debian it looks something like this.
27
28
<pre>
29
echo 'deb http://http.debian.net/debian jessie-backports main' | sudo tee -a /etc/apt/sources.list.d/jessie-backports.list
30
sudo apt-get install docker.io
31
sudo adduser $USER docker
32
# {log out & log back in}
33
groups
34
# {should include "docker"}
35
</pre>
36
37
Start up a new container with jessie, make a new user and log in as that user:
38
39
<pre>
40
docker run -it debian:jessie bash
41
apt-get update
42
apt-get install sudo
43
adduser me
44
adduser me sudo
45
sudo -u me -i
46
</pre>
47
48
h2. Install dev environment
49
50 3 Tom Clegg
<pre>
51 8 Joshua Randall
sudo apt-get install bison build-essential fuse gettext git gitolite3 graphviz \
52 7 Bryan Cosca
    iceweasel libattr1-dev libfuse-dev libcrypt-ssleay-perl libjson-perl libcurl3 libcurl3-gnutls \
53 1 Tom Clegg
    libcurl4-openssl-dev libpcre3-dev libpq-dev libpython2.7-dev \
54 8 Joshua Randall
    libreadline-dev libssl-dev libxslt1.1 linkchecker lsof nginx perl-modules \
55 10 Joshua Randall
    postgresql python python-epydoc pkg-config sudo virtualenv \
56 2 Tom Clegg
    wget xvfb zlib1g-dev
57
58 1 Tom Clegg
# ruby 2.1:
59 6 Tom Clegg
sudo apt-get install ruby2.1 ruby2.1-dev || \
60 1 Tom Clegg
(
61
 set -e
62
 mkdir -p ~/src
63
 cd ~/src
64
 wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.6.tar.gz
65
 tar xzf ruby-2.1.6.tar.gz
66
 cd ruby-2.1.6
67
 ./configure --disable-install-doc
68
 make
69
 sudo make install
70
 sudo gem install bundler
71
)
72
73
# go >= 1.3
74
sudo apt-get install golang=2:1.3.3-1 || \
75
(
76
 set -e
77
 wget https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz
78
 sudo tar -C /usr/local -xzf go1.4.2.linux-amd64.tar.gz
79
 cd /usr/local/bin
80
 sudo ln -s ../go/bin/* .
81
)
82
83
# phantomjs 1.9.8
84
(
85
 set -e
86
 PJS=phantomjs-1.9.8-linux-x86_64
87
 wget -P /tmp https://bitbucket.org/ariya/phantomjs/downloads/$PJS.tar.bz2
88
 sudo tar -C /usr/local -xjf /tmp/$PJS.tar.bz2
89
 sudo ln -s ../$PJS/bin/phantomjs /usr/local/bin/
90
)
91
</pre>
92
93
h2. Get the arvados source tree and test scripts
94
95
<pre>
96
cd
97
git clone https://github.com/curoverse/arvados.git
98
git clone https://github.com/curoverse/arvados-dev.git
99
</pre>
100
101
...or, if you're a committer with your public key on our git server:
102
103
<pre>
104
cd
105
git clone git@git.curoverse.com:arvados.git
106
git clone git@git.curoverse.com:arvados-dev.git
107
</pre>
108
109 5 Tom Clegg
h2. Start Postgres
110
111
_If you're running in a docker container_ you'll need to start Postgres manually:
112
113
<pre>
114
sudo /etc/init.d/postgresql start
115
</pre>
116
117 1 Tom Clegg
(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)
118 9 Joshua Randall
119 11 Joshua Randall
h2. Start X11
120
121
In order for the apps/workbench tests to function, firefox needs to have an X11 server (or it will fail to start). If you are on a workstation with a "real" X server, that should work. If not, Xvfb is an X server that renders to a virtual framebuffer so that selenium/firefox tests can run in headless mode. Also make sure that the DISPLAY environment variable is set accordingly.
122
123
<pre>
124
Xvfb :0.0
125
export DISPLAY=":0.0"
126
</pre>
127
128 9 Joshua Randall
h2. Setup groups
129
130
Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them. 
131 5 Tom Clegg
132 1 Tom Clegg
h2. Create a Postgres user
133
134 5 Tom Clegg
Create an "arvados" user with "create database" privileges. The test suite will create and drop the arvados_test database as needed.
135 1 Tom Clegg
136
<pre>
137
newpw=`tr -cd a-zA-Z </dev/urandom |head -c32`
138
sudo -u postgres psql -c "create user arvados with createdb encrypted password '$newpw'"
139 7 Bryan Cosca
cp -i ~/arvados/services/api/config/database.yml{.sample,} # {.example,}
140 1 Tom Clegg
newpw="$newpw" perl -pi~ -e 's/xxxxxxxx/$ENV{newpw}/' ~/arvados/services/api/config/database.yml
141
</pre>
142
143
h2. Run tests
144
145
<pre>
146
time ~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados
147
</pre>
148
149
During development, you'll probably want something more like this. It leaves its temp dirs in place after running tests, which allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on.
150
151
<pre>
152
time ~/arvados-dev/jenkins/run-tests.sh WORKSPACE=~/arvados \
153
 VENVDIR=~/.cache/arvados-venv \
154
 VENV3DIR=~/.cache/arvados-venv3 \
155
 GOPATH=~/.cache/arvados-gopath \
156
 GEMHOME=~/.cache/arvados-gemhome
157
</pre>