Arvados

• 22920-update-gem-pins @ 860cbaefd709e882e41910126358f824e5f778f3 -- developer-run-tests: #4865
• build-packages-multijob: #4807

• 22920-update-gem-pins @ 5c69a9b4a55d093fde167de7d5f83107b1044f31 -- developer-run-tests: #4867
• Is it best for our gems to declare minimum Ruby 3.0.0 to avoid problems with transitive dependencies and express that we no longer test with 2.7.0? Or leave it at 2.7.0 so it's still possible to install on 2.7.0 if you manage any transitive dependency problems yourself?
• There's an oj version pin caused by sso-provider being in arvbox, and that's not a thing any more, so I think it's quite safe to remove that version pin entirely.

(edit: nvm, see note below)¶

Rebased to main (#note-11 was accidentally based on an unmerged branch)

• 22920-update-gem-pins @ 66f2bfa3cfb8cae62a0db49bc12d15b2913c57e5
• developer-run-tests: #4870
• build-packages-multijob: #4813

• This makes our gems compatible with activesupport 7.1.5.2 (versions >=7.1.4 require 2.7.3, versions >=7.2 require Ruby 3.1)
• We can't update source:services/api/Gemfile.lock to take advantage of that until we publish an arvados gem with an updated dependency (i.e., this commit), so arvados-api-server is still at Rails 7.1.3.4
• Also unpin oj in arvados-cli -- it was pinned for compatibility with Ruby 2.4 and sso-provider, both of which are ancient history
• 22920-update-gem-pins @ 12d7965049298185b04898dad98a1b418f9a1fdc
• developer-run-tests: #4871
• build-publish-packages-python-ruby: #99 (looks like this jenkins job is a little broken atm: cannot open /home/jenkins/tmp/VENV3DIR/bin/activate: No such file)

Tom Clegg wrote in #note-11:

• Is it best for our gems to declare minimum Ruby 3.0.0 to avoid problems with transitive dependencies and express that we no longer test with 2.7.0? Or leave it at 2.7.0 so it's still possible to install on 2.7.0 if you manage any transitive dependency problems yourself?

Open to discussing this but I vote the former. I think we should allow ourselves to use features in the oldest version of interpreted languages we support. Bumping the version dependency now declares our intent even before we actually do it and prevents problems of "we used a new feature but forgot to update the metadata."

This is basically how we're handling Python. See #23097. (Python is a little better because pip install has an --ignore-requires-python option if you want to be clever.)

• There's an oj version pin caused by sso-provider being in arvbox, and that's not a thing any more, so I think it's quite safe to remove that version pin entirely.

Sounds good to me, thanks.

Tom Clegg wrote in #note-13:

• build-publish-packages-python-ruby: #99 (looks like this jenkins job is a little broken atm: cannot open /home/jenkins/tmp/VENV3DIR/bin/activate: No such file)

It was running on an unexpected node type. Updated the config, please try again.

developer-run-tests: #4872

• Update arvados-google-api-client, arvados, arvados-cli, and arvados-login-sync gems to require Ruby 3.0, update dependencies & comments accordingly
• Update arvados-google-api-client hardcoded version to 0.8.7.13
• build-publish-gem-arvados-google-api-client: #10
• build-publish-packages-python-ruby: #101

1. Publish arvados-google-api-client 0.8.7.13 from this branch
2. Publish arvados, arvados-cli, arvados-login-sync 3.2.0rc1 from this branch
3. Run bundle update again so services/api/Gemfile.lock picks up the rc gems and the third party dependency updates unblocked by the rc gems
4. Run tests and build-packages-multijob again to test against all the new dependencies
5. Merge to main

Tom Clegg wrote in #note-16:

22920-update-gem-pins @ 17afcb1565c828028af3969de0308b26c0cd68fd -- developer-run-tests: #4872

These test failures look relevant. I'll look at the branch in parallel but if you could please investigate, definitely I'd want to see these passing, or at least an explanation of how the proposed process will get us there.

Brett Smith wrote in #note-17:

These test failures look relevant. I'll look at the branch in parallel but if you could please investigate, definitely I'd want to see these passing, or at least an explanation of how the proposed process will get us there.

Yes, this is relevant. Tests passed for me locally because I didn't delete Gemfile.lock. (Is there any reason we shouldn't rm Gemfile.lock etc. before gem install in run-tests.sh?)

• Revert incompatible gem upgrade in arvados-google-api-client (retriable v2 is not compatible with v1)
• Publish arvados-google-api-client 0.8.7.14
• Update services/api/Gemfile.lock to arvados-google-api-client 0.8.7.14
• developer-run-tests: #4873
• build-packages-multijob: #4820

Tom Clegg wrote in #note-19:

22920-update-gem-pins @ 6a0ad473aec997dac50f799001460a031f9ed144

Functionally this looks pretty good. I appreciate you doing such a complete audit. I found it difficult to understand how some of the comments related to some of the pins, especially when they start talking about much older Rubies like 2.4.

What do you think about standardizing our comments on a template like:

[GEM] dropped Ruby [VERSION] in [major|minor|patch] release [VERSION].

The Ruby version mentioned should be no newer than the oldest Ruby we support. And then we have a shared understanding that the [major|minor|patch] part tells you how specific we want to make the pin to prevent future headaches. (We should document that shared understanding somewhere. I'm not sure where.)

e.g., "Rails 7.2.x releases require Ruby >= 3.1.0." becomes "Rails dropped Ruby 3.0 in minor release 7.2.0." And that explains why we pin the minor version ~> 7.1.3.

The comment can add more explanation after this if it's helpful but each one should start with this template, e.g.,

# launchy dropped Ruby 3.0 in minor release 3.1.0.
# They did not change metadata but stopped testing against it.

# securerandom dropped Ruby 3.0 in minor release 0.4.0.
# They did not announce this in the changelog or release notes.

Excepting the very weird cases like autoparse and extlib. I think we should understand that pins without comments exist because we directly depend on that API, as with retriable.

+  # public_suffix 6.0.0 dropped Ruby 2.7.
   s.add_runtime_dependency 'public_suffix', '~> 5.0'

Why aren't we updating this pin to allow at least some of the 6.x series then?

Thanks.

Brett Smith wrote in #note-21:

Functionally this looks pretty good. I appreciate you doing such a complete audit. I found it difficult to understand how some of the comments related to some of the pins, especially when they start talking about much older Rubies like 2.4.

I was trying to express the rationale for choosing a major/minor/patchlevel pin for the gems whose latest version still supports Ruby 3.0. If a gem has a history of dropping Rubies at a non-zero minor, but not at a non-zero patchlevel, then we should pin to the latest minor version.

What do you think about standardizing our comments on a template like:

[GEM] dropped Ruby [VERSION] in [major|minor|patch] release [VERSION].

That's what I was going for, except I thought "[gem] [gemversion] dropped Ruby [rubyversion]" was easier to read.

The Ruby version mentioned should be no newer than the oldest Ruby we support.

Yes, except where the gem currently supports 3.0, in which case (I think) it's still worth expressing why we chose major/minor/patchlevel, and the fact that Ruby version support is the reason for the pin.

And then we have a shared understanding that the [major|minor|patch] part tells you how specific we want to make the pin to prevent future headaches. (We should document that shared understanding somewhere. I'm not sure where.)

Same.

e.g., "Rails 7.2.x releases require Ruby >= 3.1.0." becomes "Rails dropped Ruby 3.0 in minor release 7.2.0." And that explains why we pin the minor version ~> 7.1.3.

I've fixed this one to use the same format as the others.

Excepting the very weird cases like autoparse and extlib. I think we should understand that pins without comments exist because we directly depend on that API, as with retriable.

I think it's worth noting known API incompatibilities, because it's quite common for major releases to remain API-compatible. Arguably, the ideal gem maintains API compatibility forever but increases the major version when dropping support for a Ruby version. If we note known API changes, we can skip re-learning that when looking for API-compatible updates, e.g., when we no longer support Ruby 3.0.

Why aren't we updating this pin to allow at least some of the 6.x series then?

Oops, fixed.

• developer-run-tests: #4874 (lib/crunchrun failure is an unrelated flaky test)
• build-packages-multijob: #4821

I get the sense that we're not 100% aligned on how to handle the specific case of "gem has dropped support for one of our Rubies in the past but currently supports all our current Rubies" and I can't tell if we're misaligned on how we should actually pin it or just how we should document it. Let me talk it through with the understanding that it could be either.

That's what I was going for, except I thought "[gem] [gemversion] dropped Ruby [rubyversion]" was easier to read.

I'm not too fussed about the word order, that's fine, but I do feel like the explicit [major|minor|patch] callout helps explain why the pin is written a certain way. In other words, it addresses this:

Yes, except where the gem currently supports 3.0, in which case (I think) it's still worth expressing why we chose major/minor/patchlevel, and the fact that Ruby version support is the reason for the pin.

You're basically already spelling it out any time it's necessary so it seems like it might as well become part of the template. At least for minor or patch releases. For example:

+  # faraday 2.13.4 supported Ruby 3.0. Support for Ruby 2.7 was
+  # dropped in a minor version (2.9.0).

Could just become "faraday minor version 2.9.0 dropped Ruby 2.7." And then we just have a shared understanding that we're pinning ~> 2.13.0 because at the time that was the latest version that supported all of our Rubies. Ditto for all the changed comments in services/api/Gemfile.

To get really concrete, with the lack of callout, there are some pins that, given the comment, I would expect a different pin. And so this is where I'm wondering about the misalignment.

+  # googleauth 1.14.0 dropped Ruby 2.
+  s.add_runtime_dependency 'googleauth', '~> 1.15'

Since apparently a minor version dropped a Ruby, I would expect to pin the minor version ~> 1.15.0. Analogously here:

+  # kramdown 2.5.0 dropped Ruby 2.4.
+  s.add_development_dependency 'kramdown', '>= 1.5', '< 3'

I've fixed this one to use the same format as the others.

I am not seeing this change in the branch as pushed.

• kramdown < 2.6. I had < 3.0 before, because although it dropped 2.4 in a minor release, it still supports Rubies so old even we don't care about them, and I figured it will probably continue to do so. Also, it's only a development dependency. But yes, avoiding 2.6 is more cautious and consistent.
• googleauth < 1.16.

Added a comment about pinning minor versions when previous minor versions have dropped support for a Ruby version.

Added "gem x.y is the highest minor version we've tested" comments in cases where the only reason for a minor version pin is that the gem has previously dropped a Ruby at a minor version.

I am not seeing this change in the branch as pushed.

Oops, actually pushed my latest commit this time.

• 22920-update-gem-pins @ bebed8166daf59968529d606ee15d3e96996101c
• developer-run-tests: #4875 (unrelated wb failure)
• build-packages-multijob: #4822
• retry rocky8 build-packages-rocky8: #1155

• 621774c27a11df6f4a095acb696a73727f327b8e

build-publish-packages-python-ruby: #102 failed. The warning about stringio is probably not causing the failure. (The same warning appears when building the gems, which succeeds anyway.)

======= Start upload ruby gems
2025-09-03 16:58:33 arvados-dev.upload.gems[1497] INFO: Uploading /tmp/workspace/build-publish-packages-python-ruby/sdk/ruby/arvados-3.2.0rc1.gem
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      stringio (>= 0)
      Available/installed versions of this gem:
      - 3.1.5
      - 3.0.1.2
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.
Traceback (most recent call last):
  File "/tmp/workspace/build-publish-packages-python-ruby@tmp/tmp.OdB0FASjmC/jenkins/run_upload_packages.py", line 356, in <module>
    main(sys.argv[1:])
  File "/tmp/workspace/build-publish-packages-python-ruby@tmp/tmp.OdB0FASjmC/jenkins/run_upload_packages.py", line 352, in main
    build_suite_and_upload(target, last_upload_ts, args)
  File "/tmp/workspace/build-publish-packages-python-ruby@tmp/tmp.OdB0FASjmC/jenkins/run_upload_packages.py", line 342, in build_suite_and_upload
    suite.update_packages(since_timestamp)
  File "/tmp/workspace/build-publish-packages-python-ruby@tmp/tmp.OdB0FASjmC/jenkins/run_upload_packages.py", line 115, in update_packages
    self.upload_files(upload_paths)
  File "/tmp/workspace/build-publish-packages-python-ruby@tmp/tmp.OdB0FASjmC/jenkins/run_upload_packages.py", line 107, in upload_files
    self.upload_file(path)
  File "/tmp/workspace/build-publish-packages-python-ruby@tmp/tmp.OdB0FASjmC/jenkins/run_upload_packages.py", line 160, in upload_file
    raise subprocess.CalledProcessError(push_returncode, cmd)
subprocess.CalledProcessError: Command '['gem', 'push', '/tmp/workspace/build-publish-packages-python-ruby/sdk/ruby/arvados-3.2.0rc1.gem']' returned non-zero exit status 1.
======= upload ruby gems -- FAILED

Tom Clegg wrote in #note-16:

1. Publish arvados, arvados-cli, arvados-login-sync 3.2.0rc1 from this branch
2. Run bundle update again so services/api/Gemfile.lock picks up the rc gems and the third party dependency updates unblocked by the rc gems
3. Run tests and build-packages-multijob again to test against all the new dependencies

22920-update-gem-pins @ 079990b7f0956ec52b8092537b55f9da0d13d1c9
developer-run-tests: #4876
build-packages-multijob: #4826