Arvados

This text was originally written before we had the build-docker-image.yml playbook, and instead we had a separate Dockerfile for each distribution we built packages for. While the proposed solution is less relevant today, the problem may still exist: there's no way to control what resources are used during compilation.

Original description¶

Our build Dockerfiles have a bunch of places where they use the CPU count for build parallelism:

    /usr/local/rvm/bin/rvm install 2.7 -j $(grep -c processor /proc/cpuinfo) --disable-binary
RUN /usr/local/rvm/bin/rvm-exec default bundle config --global jobs $(let a=$(grep -c processor /proc/cpuinfo )-1; echo $a)
ENV MAKE "make --jobs $(grep -c processor /proc/cpuinfo)" 

A few ways this isn't ideal:

• The current process may not be allowed to use all CPUs. It would be more accurate to check scheduler affinity and use a number from that.
• The user may want a different number for various reasons (e.g., I want to leave a core or two open for development work while the build runs in the background).
• It's not DRY.

What I would like is:

• The build Dockerfiles take a JOBS argument. It can default to some reasonable static number like 8.
• The build scripts take a --jobs/-j argument to let the user specify this for all Docker builds. It can default to a number from the scheduler.