Arvados

The release should cherry-pick these small bugfixes pushed directly to main:

• 3f41fa3decdcca7e202876c3510469f17b390e55
• 75e79dd14577aa89527c514f055a1baf331937e9
• 2300891524b1ca0573d7614c68ecd650075c3338
• 824e3f459a4c97eb8dd5ad3a5e2e66984b642603

Adding to the list:

• 0172f9f49f918f57f478b0667fc44eeff4feffce
• 509294706a6e71ac0a6616820e5a0c30298bc04f
• 1972f29f638f8962bfa8f685eaebc0f9b03fd6b2

These are sufficient to get Ubuntu 20.04 working again: test-provision-ubuntu2004: #1159

Brett Smith wrote in #note-31:

• 2300891524b1ca0573d7614c68ecd650075c3338

This commit surfaced #22751. It is the least important commit, a one-character fix to an undocumented build process, so I'm removing it from the branch. The rest remain.

Brett Smith wrote in #note-37:

3.1-staging @ abc8d2564d5768637f52cc9e5db136aa1cd17d2c

developer-run-tests: #4722

developer-run-tests-doc-sdk-java-R: #2233

LGTM

build-and-publish-rc-packages: #267

docker-jobs-image-release: #85

test-provision: #1151 - Everything is passing except debian11 which seems to be hitting some transient failure, none of this is about Arvados:

----------
          ID: docker-software-package-install-deps
    Function: pkg.installed
        Name: python3-pip
      Result: False
     Comment: Problem encountered installing package(s). Additional info follows:

              errors:
                  - Running scope as unit: run-r79483305a13b45efa43290f3e178f1b2.scope
                    E: Failed to fetch https://deb.debian.org/debian/pool/main/p/python-pip/python-pip-whl_20.3.4-4%2bdeb11u1_all.deb  Error reading from server - read (5: Input/output error) [IP: 151.101.2.132 443]
                    E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
     Started: 21:16:29.692747
    Duration: 2424.556 ms
     Changes:   
  Name: nginx - Function: service.mod_watch - Result: Changed - Started: 21:16:32.119442 - Duration: 54.004 ms

Will retry in the morning.

Unfortunately almost forgot #22700 so here we go again. df84bbc0a512cd7ca7b9c772a51b8e9e42387cd9

developer-run-tests: #4723

developer-run-tests-doc-sdk-java-R: #2234

3.1.1~rc2-1

build-and-publish-rc-packages: #271

docker-jobs-image-release: #87

test-provision: #1153

arvados-cwl-conformance-tests: #1910

Test pipeline pirca-xvhdp-f68qx7rob3p65k0

The error feels like it could be transient so retrying at pirca-xvhdp-cv6l0mpe0vz4541

Manual testing: spun up a new Ubuntu 20.04 cloud node. Installed arvados-api-server with a bad database configuration and got the expected error. Fixed the config and re-installed successfully. Set up controller and was able to login remotely and use that token for work. Ran gem install --user arvados-cli and confirmed it works (because the key fix is in arvados-google-api-client which already got published).

I am going to hope really hard that with more time passing we will have better pipeline luck: pirca-xvhdp-oq0ad2a69kca0lv

3.1-staging @ e1f15f162c91a56818337de2882958f2e7ea57c0

developer-run-tests: #4729

developer-run-tests-doc-sdk-java-R: #2235

WGS workflow: tordo-xvhdp-qk0de7b3x8xvdwn

S3 workflow: tordo-xvhdp-8sy7xbgghw4sp1a

Draft release notes have been updated.

build-and-publish-rc-packages: #272

docker-jobs-image-release: #88

test-provision: #1158 - ubuntu2204 lost a race condition, retry test-provision-ubuntu2204: #306

packer-build-compute-image: #324 - this revealed #22775

After merging #22775, I am making the call to pick up where we left off, rather than start the whole release process over. The single line changed does not go into any package or affect any build other than the compute node image. Starting over would be hours of human time, and dozens of hours of compute time, basically just to check boxes.

New compute image packer-build-compute-image: #325

CWL conformance tests arvados-cwl-conformance-tests: #1916

pirca updated (manually did a dist-upgrade on API to work around #22766), health and diagnostics checks pass

pirca WGS run: pirca-xvhdp-jutozyobkgiffy3

2025-04-11T21:38:54.311435778Z crunch-run 3.1.1~rc3 (go1.23.6) started
2025-04-11T21:38:59.048900753Z Using FUSE mount: /usr/bin/arv-mount 3.1.1rc3
2025-04-11T21:38:59.055184775Z Using container runtime: docker Engine 27.5.1, containerd 1.7.27, runc 1.2.5, docker-init 0.19.0
2025-04-11T21:39:15.107210230Z INFO /usr/lib/python3-arvados-cwl-runner/bin/arvados-cwl-runner 3.1.1rc3, arvados-python-client 3.1.1rc3, cwltool 3.1.20240508115724

Manual testing: between my manual testing in #note-47 and #22775#note-1 I believe we're covered.

Brett Smith wrote in #note-53:

Manual testing: between my manual testing in #note-47 and #22775#note-1 I believe we're covered.

Also confirmed in the logs there were zero broken compute nodes while the pirca WGS workflow was running.

Late last night I got the idea we should test a CUDA workflow too, it passed with all the correct versions: pirca-xvhdp-vjqsp1s9w2jkqj4

3.1-staging c5053c929b57c1f73bfb2e8189066872fc96b6f2

and release notes commit:d679414fd63049ce70029420b7935f868055d404

reviewed and approved

Upgrade notes and version commit 07c0177a743e5c531c1cd848eb3f1a36c0f0abb0

build-and-publish-rc-packages: #273

build-publish-packages-python-ruby: #96 (both Python and Ruby, oops, but no harm done for this particular release)

docker-jobs-image-release: #89

publish-packages-to-stable-repo: #291

build-java-sdk: #59

build-package-r: #39

build-and-release-docker-arvbox-demo-image: #64

Jenkins test configuration is unchanged since 3.1.0. See #22510#note-41.