Arvados

I have been trying different approaches and am somewhat blocked.

For what I've seen, there're 2 main cases when building packages:

1. New version release: the package version is provided by using an env variable and it's the same on all packages.
2. New dev package: the package version is built from the last tag (aka: release version) plus a timestamp. This timestamp may or may not be different across packages for every single commit.

The issue with changing timestamps is that I haven't found a clean way to get (for example) the arvados-python-client version from the arvados-cwl-runner package build process. I think we are using different timestamps to avoid creating/uploading new dev packages when components don't change from one commit to another.

Talking to Fernando about this, I learned that we're no longer uploading dev packages to PyPI, so maybe it won't be a big issue to start using the same timestamp on every component, so that every package build process knows that its own version matches the other packages versions.

If we should keep creating different timestamps for every dev package, maybe I should change the entire approach and have a completely external process that gets all the versions creating some kind of queryable common index?

Lucas Di Pentima wrote:

I have been trying different approaches and am somewhat blocked.

For what I've seen, there're 2 main cases when building packages:

1. New version release: the package version is provided by using an env variable and it's the same on all packages.
2. New dev package: the package version is built from the last tag (aka: release version) plus a timestamp. This timestamp may or may not be different across packages for every single commit.

The issue with changing timestamps is that I haven't found a clean way to get (for example) the arvados-python-client version from the arvados-cwl-runner package build process. I think we are using different timestamps to avoid creating/uploading new dev packages when components don't change from one commit to another.

Talking to Fernando about this, I learned that we're no longer uploading dev packages to PyPI, so maybe it won't be a big issue to start using the same timestamp on every component, so that every package build process knows that its own version matches the other packages versions.

If we should keep creating different timestamps for every dev package, maybe I should change the entire approach and have a completely external process that gets all the versions creating some kind of queryable common index?

The most important is (1) so that installing from PyPi doesn't fetch incorrect dependencies (e.g. installing arvados-cwl-runner 1.3 doesn't fetch arvados-python-client 1.4)

For (2) perhaps for a-c-r or arvados-fuse version "X" the dependency is on "arvados-python-client is <= X" ? Then the default behavior is to get the latest that is equal or lesser than the version of the package being built. (Assuming PyPi doesn't complain if version X doesn't exist).

Updates at 3bc30ad82 - branch 15209-python-arv-deps-pinned

Ok, so after some bash script fiddling I think I got this working. Thanks Peter for your suggestions, they worked perfectly.

The python-arvados-client sdist package is built even if --only-build ask for another package, so it's available when other dependant packages are installed on their virtualenv. Pass -f <pysdk_sdist_path> to pip so that it doesn't try to download arvados-python-client from PyPI's index.

For testing, I ran:

$ WORKSPACE=$(pwd) ./build/run-build-packages-one-target.sh --test-packages --build-version "1.4.1-1" 

Also, installed arvados-cwl-runner's deb package and ran some example cwl workflow against 4xphq with both the fake 1.4.1 release version and also the dev version. In both cases, versions were correctly reported, for example:

lucas@veritas:~/cwl-examples$ arvados-cwl-runner --submit 1st-tool.cwl echo-job.yml 
INFO /usr/bin/arvados-cwl-runner 1.4.1, arvados-python-client 1.4.1, cwltool 1.0.20190607183319
INFO Resolved '1st-tool.cwl' to 'file:///home/lucas/cwl-examples/1st-tool.cwl'
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using collection cache size 256 MiB
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO [container 1st-tool.cwl] 4xphq-xvhdp-744yc2e7n7iofay state is Committed
INFO [container 1st-tool.cwl] 4xphq-xvhdp-744yc2e7n7iofay is Final
INFO Overall process status is success
INFO Final output collection 282fbb8a925551343e98e762fa1ec1be+57 "Output of 1st-tool.cwl" (4xphq-4zz18-haoisak7o1k6kuj)
{}
INFO Final process status is success

Lucas Di Pentima wrote:

For testing, I ran:

[...]

Also, installed arvados-cwl-runner's deb package and ran some example cwl workflow against 4xphq with both the fake 1.4.1 release version and also the dev version. In both cases, versions were correctly reported, for example:

[...]

This LGTM.

Fixed some silly mistakes at 4139c5dfd
Test run: https://ci.curoverse.com/job/developer-run-tests/1352/