Hacking prerequisites » History » Revision 83
Revision 82 (Brett Smith, 01/28/2025 07:38 PM) → Revision 83/91 (Brett Smith, 02/13/2025 04:21 PM)
{{>toc}}
h1. Hacking prerequisites
This page describes how to install all the software necessary to develop The Arvados and test suite can run tests. in a Docker container, a VM, or your workstation -- provided a few prerequisites are satisfied.
h2. Host options
You must have h3. Starting on your workstation
If your workstation is a debian buster system running -- and you don't mind installing a supported distribution. That system can be installed directly bunch of packages on hardware; running on a cloud instance; or in a virtual machine.
h3. Supported distributions
As your workstation, some of February 2025/Arvados 3.0, these instructions and them without apt -- the entire test suite are known easiest way to work on Debian 11 "bullseye." They mostly work on Debian 12 "bookworm," but a small handful of tests fail because they haven't been adapted get running is to newer software yet.
You may try to run these instructions and tests on Ubuntu 20.04 "focal"/22.04 "jammy"/24.04 "noble," but they have not been tested and you are likely bare metal. Skip to hit some bugs throughout. "Dependencies".
These instructions are not suitable for any Red Hat-based distribution and many tests would fail on them.
h3. Base configuration
On your development system, you Other linux distributions should have a user account work too with full permission some modifications, but it's probably easier to use sudo. a VM.
You can h3. Starting on a VM
Another option is to create a virtual machine using something like Xen or VirtualBox, and run the Ansible playbook to install your development system debian buster on a different system. To do this, it. The instructions below assume you must have permission to just a few basic requirements:
* SSH into your server
* sudo (@apt-get install sudo@)
* A user account from the system running Ansible (the "control node") to the development system you're installing (the "target node"). with sudo privileges
h3. Virtual machine requirements Starting in a docker container
If _[[Arvbox]] provides a preinstalled Docker-based dev environment. The following instructions are for creating a dev environment inside Docker from scratch._
This can get you run your development system in a virtual machine, it needs some permissions. Many environments will allow these operations by default, started quickly, but they could be limited by (unlike the above options) you'll need to remember to use something like @docker commit@ to save your virtual machine setup. state before shutting down your container.
* It must be able to create and manage FUSE mounts (@/dev/fuse@) See http://docker.io for more about installing docker. On debian it looks something like this.
<pre>
* It must be able to create sudo apt-get install docker-ce
sudo adduser $USER docker
# {log out & log back in}
groups
# {should include "docker"}
</pre>
Start up a new container with debian 10 (buster), make a new user and run Docker containers log in as that user:
<pre>
* It must be able to create and docker run Singularity containers—this requires creating and managing block loopback devices (@/dev/block-loop@) -it --privileged debian:10 bash
* It must have the @fs.inotify.max_user_watches@ sysctl set apt-get update
apt-get -y install sudo
adduser me
adduser me sudo
sudo -u me -i
</pre>
The "--privileged" is required in order for /dev/fuse to at least 524288. Our Ansible playbook be accessible (without it, no tests that require FUSE will try to set this on the managed host, but if it is unable to do so, you may need to set it on the parent host instead. work).
h2. Install development dev environment with
h3. With Ansible
h3. This is a prototype still in development, but initial development was done in January 2025 and it can automate much of the process for you.
h4. Install Ansible
The simplest thing to do is @pip install@ Ansible inside a virtualenv:
<pre><code class="shell">sudo apt install python3-venv
python3 -m venv ~/arvados-ansible
~/arvados-ansible/bin/pip install "ansible~=8.7"
</code></pre>
If that works, you're done and you can go on to the next section. There are "other installation options":https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html if you can't do this for some reason. For background:
For background, the * The Arvados Ansible playbooks are tested with Ansible 8, which we use for its Python version compatibility. Older versions are known not to work. You're welcome to try newer versions if you want to for any reason, but they haven't been tested.
h3. * There are "other installation options":https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html if you can't do this for some reason.
h4. Write an Arvados database configuration
Next, write an Arvados configuration file that defines how you want to set up the PostgreSQL database. Write a file @~/arvados-ansible/zzzzz.yml@ like this:
<pre><code class="yaml">Clusters:
zzzzz:
PostgreSQL:
Connection:
user: arvados
password: GoodPasswordHere
dbname: arvados_test arvados_development
host: localhost
port: "5432"
</code></pre>
The cluster ID *must* must be @zzzzz@. You can change the @user@, @password@, and @dbname@ settings freely. Our Ansible playbook will configure PostgreSQL so your settings here work.
The playbook will always install the @postgresql@ server package. It will *not* change any PostgreSQL configuration except If you already have this installed and have configured it to add @pg_hba.conf@ entries for this user. You should only change listen somewhere other than the default, you may update @host@ and @port@ if you need to use a PostgreSQL server reflect that. Note that @port@ is already installed and running somewhere else.
a string containing digits.
h4. Write an Ansible inventory
An inventory file tells Ansible what host(s) to manage, how to connect to them, and what settings they use. Write an inventory file to @~/arvados-ansible/inventory.ini@ like this:
<pre><code class="ini">[arvados-test]
# This is the list of host(s) where we're installing the test environment.
# The line below installs on the same system running Ansible.
# If you want to manage remote hosts, you can write your own host list:
# <https://docs.ansible.com/ansible/latest/getting_started/get_started_inventory.html>
localhost ansible_connection=local
[arvados-test:vars]
# The path to the Arvados cluster configuration you wrote in the previous section.
arvados_config_file={{ lookup('env', 'HOME') }}/arvados-ansible/zzzzz.yml
# The primary user doing Arvados development and tests.
# This user will be added to the `docker` group.
# It defaults to the name of the user running `ansible-playbook`.
# If you want to configure a different user, set that here:
#arvados_dev_user=USERNAME
# The authentication mechanism to allow in `pg_hba.conf`.
# The default is `scram-sha-256`, which is the most secure method on the most
# recent versions of PostgreSQL.
# If your development system is running Debian 11, set this to `md5` here.
#arvados_postgresql_hba_method=md5
</code></pre>
h4. Run the playbook
The playbook is available from the @tools/ansible@ directory of the Arvados source tree. If you don't already have a checkout of the Arvados source, get it by running:
<pre><code class="sh">git clone https://git.arvados.org/arvados.git ~/arvados</code></pre>
The basic command to run the playbook is:
<pre><code class="sh">cd arvados/tools/ansible
~/arvados-ansible/bin/ansible-playbook -K -i ~/arvados-ansible/inventory.ini -K install-test-env.yml</code></pre>
@ansible-playbook@ has many options to control how it runs that you can add if you like. Refer to "the @ansible-playbook@ documentation":https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html for more information.
After the playbook runs successfully, you should be able to run the Arvados tests from a source checkout on your development host. e.g.,
<pre><code class="sh">cd arvados
WORKSPACE="$PWD" build/run-tests.sh --temp ~/arvados-test --interactive
</code></pre>
Refer to [[Running tests]] for details.
h4. Troubleshooting
The playbook writes your database configuration at @~/.config/arvados/config.yml@ and sets up a hook @/etc/profile.d/arvados-test.sh@ to set your @CONFIGSRC@ environment variable to that directory. If most tests fail with a database connection error, check that this variable is set:
<pre>$ echo "${CONFIGSRC:-UNSET}"
/home/you/.config/arvados
</pre>
If that reports @UNSET@, add a line to set @CONFIGSRC="$HOME/.config/arvados"@ to your shell configuration, or set it manually when you run @run-tests.sh@:
<pre><code class="sh">WORKSPACE="$PWD" CONFIGSRC="$HOME/.config/arvados" build/run-tests.sh ...
</code></pre>
h4. Notes
The playbook will install symlinks for Go, Node, Singularity, and Yarn under @/usr/local/bin@. The actual tools are installed under @/opt@. If you need different versions of these tools for other work on this system, work, you'll need to customize your @PATH@ environment variable so the Arvados versions are found first when you're doing Arvados work.
h3. Manually
Start with Debian 10+ and run the following commands as root.
Note that the last command here ("arvados-server install -type test") installs additional debian packages to your system, along with additional software in /var/lib/arvados/ (such as suitable versions of Ruby and Go) that do not interfere with system packages. It also creates a postgresql database user named "arvados" with an insecure password. Don't expose this postgresql server to the internet or to untrusted users!
<pre>
apt update
apt install wget ca-certificates
wget https://apt.arvados.org/bullseye/pool/main/a/arvados-server/arvados-server_2.4.3-1_amd64.deb
dpkg -i arvados-server_2.4.3-1_amd64.deb
arvados-server install -type test
</pre>
Alternatively, install Go ≥ 1.20 (see https://golang.org) and git, and run arvados-server from source.
<pre>
wget -O- https://go.dev/dl/go1.22.1.linux-amd64.tar.gz | tar -C /usr/local -xzf -
ln -s /usr/local/go/bin/* /usr/local/bin/
apt install git build-essential libpam-dev
cd
git clone https://git.arvados.org/arvados.git
cd arvados
go mod download
go run ./cmd/arvados-server install -type test
</pre>
h2. Start Postgres
_If you're running in a docker container_ you'll need to start Postgres manually:
<pre>
sudo /etc/init.d/postgresql start
</pre>
(If you're on a regular workstation/server/VM, startup scripts have already taken care of that for you.)
h2. Setup groups
Make sure the fuse and docker groups exist (create them if necessary) and that the user who will run the tests is a member of them.
h2. Run tests
<pre>
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados
</pre>
During development, you'll probably want something more like this. It reuses the given temp directory, which avoids a lot of repetitive downloading of dependencies, and allows you to save time with @--skip-install@ or @--only-install sdk/ruby@ and so on.
<pre>
mkdir -p ~/.cache/arvados-build
time ~/arvados/build/run-tests.sh WORKSPACE=~/arvados --temp ~/.cache/arvados-build
</pre>