Arvados

here is an example of a bad run

https://ci.curoverse.com/view/Arvados%20build%20pipeline/job/build-packages/1232/console

So at least part of the problem is that we're overriding the Docker image's own command. The appropriate Software Collections need to be enabled before running the package build script. We may need to integrate those smarts into our build wrapper.

I ran this locally and it looks like I successfully generated a lot of debian7, debian8, and ubuntu1204 packages, but this seems to have prevented centos6 from working (oops, scrollback buffer not long enough to go back further):

Step 0 : FROM centos:6
 ---> a005304e4e74
[...]
Step 5 : RUN gpg --keyserver pool.sks-keyservers.net --recv-keys D39DC0E3 &&     curl -L https://get.rvm.io | bash -s stable &&     /usr/local/rvm/b
in/rvm install 2.1 &&     /usr/local/rvm/bin/rvm alias create default ruby-2.1 &&     /usr/local/rvm/bin/rvm-exec default gem install fpm
 ---> Using cache
 ---> 6c5c74a464cb
Step 6 : ADD https://www.softwarecollections.org/en/scls/rhscl/rh-python34/epel-6-x86_64/download/rhscl-rh-python34-epel-6-x86_64.noarch.rpm https:/
/www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm /opt/
 ---> Using cache
 ---> 1f1e21be1d5b
Step 7 : RUN yum -q -y install /opt/rhscl-*.rpm
 ---> Running in 85946f117e99
Error: Nothing to do
2015/07/23 13:43:36 The command [/bin/sh -c yum -q -y install /opt/rhscl-*.rpm] returned a non-zero code: 1
debian8
[...]
Successfully built 3a1f88c64db1

real    21m7.939s
user    0m1.240s
sys     0m10.172s
Unable to find image 'arvados/build:centos6' locally
Pulling repository arvados/build
2015/07/23 12:21:07 HTTP code: 404
ERROR: centos6 build failed with exit status 1.
/jenkins/run-build-packages.sh is running from /jenkins
Workspace is /arvados
xargs: chmod: Text file busy
umask is 0022

The previous "ADD" step seems not to have achieved the desired result:

$ docker run -it 1f1e21be1d5b
bash-4.1# find /opt -ls
    60    4 drwxr-xr-x   3 root     root         4096 Jul 23 16:03 /opt
    62    4 drwxr-xr-x   2 root     root         4096 Mar 26 12:22 /opt/rh

I didn't (yet) get any further than this debugging.

Tom Clegg wrote:

The previous "ADD" step seems not to have achieved the desired result:

For me, using Docker 1.6.0, these steps look like:

 ---> c4fcc26894d5
Removing intermediate container 288ac9ea4402
Step 6 : ADD https://www.softwarecollections.org/en/scls/rhscl/rh-python34/epel-6-x86_64/download/rhscl-rh-python34-epel-6-x86_64.noarch.rpm https://www.softwarecollections.org/en/scls/rhscl/python27/epel-6-x86_64/download/rhscl-python27-epel-6-x86_64.noarch.rpm /opt/
Downloading [==================================================>]  3.08 kB/3.08 kB
Downloading [==================================================>] 3.096 kB/3.096 kB
 ---> 9163d8a5647f
Removing intermediate container eb6f020f12ba
Step 7 : RUN yum -q -y install /opt/rhscl-*.rpm
 ---> Running in 10009d0ac129
 ---> dd4163193c40
Removing intermediate container 10009d0ac129

ci is also using Docker 1.6.0. Please upgrade and try again.

Indeed, with docker 1.6 everything works fine. Tried installing the centos6 packages, which worked fine.

I noticed the Workbench and API packages don't seem to have dependencies encoded (at least in centos6). AIUI that would be a separate feature request, not a bug introduced here. I haven't actually installed all of the packages on all the OS to make sure they work. Unless you think I should actually do that, LGTM. Thanks.

Tom Clegg wrote:

I noticed the Workbench and API packages don't seem to have dependencies encoded (at least in centos6). AIUI that would be a separate feature request, not a bug introduced here. I haven't actually installed all of the packages on all the OS to make sure they work. Unless you think I should actually do that, LGTM. Thanks.

This is sort of how they're designed to work: the packages just include source code, and then you actually get their dependencies and run them with Bundler. It's goofy, but at least it's consistent with our documentation.

I'll merge the branch. Thanks.