Arvados

Some work in progress on 13823-bionic.

easy_install3 has disappeared, and we're supposed to use "pip3 install" instead. This will be easy to migrate if it turns out "pip install" and "pip3 install" work for all older releases, too. Testing that.

With the pip3 change I can build bionic packages, but the python packages have dependency problems.

"Depends: python-ws4py (< 0.4) but 0.4.2+dfsg1-2 is to be installed"

(python-arvados-python-client depends on ws4py<0.4 because tests failed with 0.4.1 -- but ubuntu:bionic ships 0.4.2)

SDK tests seem to pass with ws4py 0.4.2.

Same problem with ruamel.yaml: bionic ships 0.15.34-1 but sdk/python requires <0.15.

python-arvados-python-client : Depends: python-ruamel.yaml (< 0.15) but 0.15.34-1 is to be installed

d015f8bd07c4890b3084306a6cdd67aeda7f81f0 introduced the <0.15 part; #13435 suggests we copied it from cwltool, which has since changed to <0.16 in 7f8a62c7b4f63480fb187fc7def4c6e3695e669c.

13823-bionic @ f8f3944e6ba1f470eedf48aa609a50c780fe4b6d

...successfully produces packages for all distros on my dev box. Haven't tried test-packages.

• switch from cure-fpm to fpm
• upgrade ws4py and ruamel.yaml python deps

I reviewed and made some more improvements on this branch, latest at bc97d806fa1fb15e49cefe819f00089ebc33d376

• fixed package testing for bionic
• merged latest master

From my perspective this is now ready for merging. Then:

• add the build steps to CI to build, upload and publish the packages in -dev
• do some functional testing (i.e. do the packages actually create a working version of Arvados?) on a bionic system
• update the https://doc.arvados.org/install/install-manual-prerequisites.html page to list bionic as a supported distribution

This was merged, and the necessary CI changes have been made. Packages have been published to the bionic-dev repo, and will be going forward.

Next step is a functional test.

This was tagged for the upcoming release, but I can't tell if it's done or not. Merged, but needs to be tested still?

Tom Morris wrote:

This was tagged for the upcoming release, but I can't tell if it's done or not. Merged, but needs to be tested still?

Everything is done in this ticket, but there is a known issue with a cwltool/ruamel version incompatibility with bionic, which is being handled in #14483. Once that is merged, this should work, and we can test our 18.04 packaging.

Well, the ruamel issue is resolved, but there's one more dependency issue, now with python-avro:

root@1aa90ea77c25:/etc/apt/sources.list.d# apt-get install python-schema-salad
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python-schema-salad : Depends: python-avro (< 1.8.2) but 1.8.2+dfsg-1 is to be installed
E: Unable to correct problems, you have held broken packages.
root@1aa90ea77c25:/etc/apt/sources.list.d# apt-cache policy python-avro
python-avro:
  Installed: (none)
  Candidate: 1.8.2+dfsg-1
  Version table:
     1.8.2+dfsg-1 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
     1.8.1-2 500
        500 http://apt.arvados.org bionic-dev/main amd64 Packages