Idea #23000
closed
Figure out how to consistently target a platform
Description
We recently tried to upgrade services/api/Gemfile.lock to use nokogiri 1.18. When we did that, during the package build, apparently where we run bundle cache, we got this error:
Because every version of rails depends on actionpack = 7.1.3.4 and every version of actionpack depends on rails-html-sanitizer ~> 1.6, every version of rails requires rails-html-sanitizer ~> 1.6. And because every version of rails-html-sanitizer depends on nokogiri >= 1.15.7, != 1.16.0.rc1, != 1.16.0, != 1.16.1, != 1.16.2, != 1.16.3, != 1.16.4, != 1.16.5, != 1.16.6, != 1.16.7, every version of rails requires nokogiri >= 1.15.7, != 1.16.0.rc1, != 1.16.0, != 1.16.1, != 1.16.2, != 1.16.3, != 1.16.4, != 1.16.5, != 1.16.6, != 1.16.7. So, because nokogiri >= 1.15.7, != 1.16.0.rc1, != 1.16.0, != 1.16.1, != 1.16.2, != 1.16.3, != 1.16.4, != 1.16.5, != 1.16.6, != 1.16.7 could not be found in cached gems or installed locally for any resolution platforms (ruby) and Gemfile depends on rails ~> 7.1.3.4, version solving has failed. The source contains the following gems matching 'nokogiri (>= 1.15.7, != 1.16.0.rc1, != 1.16.0, != 1.16.1, != 1.16.2, != 1.16.3, != 1.16.4, != 1.16.5, != 1.16.6, != 1.16.7)': * nokogiri-1.15.7-x86_64-linux * nokogiri-1.18.8-x86_64-linux ERROR: arvados-api-server package prep failed
I tried to fix this by running gem fetch --platform=ruby in the previous step. Since that's the platform named the parentheses, I figured that was the platform that Bundler was targeting. This got the package build passing, but then on install the package tests got this simple error:
Could not find nokogiri-1.15.7 in locally installed gems Install missing gems with `bundle install`. ERROR: arvados-api-server test on arvados/package-test:ubuntu2204 failed with exit status 1
(Yes, the version is different, because in this same branch I also downgraded to restore Ruby 2.7 support. I am confident the gem fetch --platform change caused this error.) This error presumably comes from a bundle command in the Rails package postinst, but unfortunately there's no more output to suggest which one. I would not assume that it must be bundle install or the bundle config we run before that. In my experience, different Bundler subcommands look at the gemset in different ways. It could be the later bundle exec true.
It might be worth looking at this after we drop Ruby 2.7 and upgrade to Bundler 2.5. There might be bugfixes that help smooth this over, and there's no immediate issue, so it can wait.
Updated by Brett Smith 9 months ago
- Related to Bug #22998: build-packages-debian12 (and others) failing after nokogiri dependency update added
Updated by Brett Smith 9 months ago
- Follows Support #22922: Upgrade bundler to 2.5.x throughout added
Updated by Brett Smith 9 months ago
- Blocks Feature #23015: Support RHEL 10 added
Updated by Brett Smith 9 months ago
https://github.com/rubygems/rubygems/issues/5743#issuecomment-1189487877 suggests that this might be related to the fact that we have ruby listed under PLATFORMS in our Gemfile.lock. This is apparently not what modern Bundler is expecting. Replacing it with x86_64-linux might be sufficient.
Note that PLATFORMS declares all the platforms where you intend to deploy this application. It does not limit developers to working on a particular platform. It only means that when Bundler updates gems, it will make sure that all gems are available for all platforms you declare you need.
Updated by Brett Smith 4 months ago
- Target version set to Development 2025-11-26
- Assigned To set to Brett Smith
- Status changed from New to In Progress
- Due date deleted (
08/29/2025)
Updated by Brett Smith 4 months ago
· Edited
23000-gemfile-platform-fixes @ 3c7eafdd0f9b4b67773744c717726ba23a9f7644
build-packages-multijob: #4936
Working backwards: the best way to fix #23294 is to force it to use the "Ruby" platform so we avoid the prebuilt gem that requires too-new glibc. This branch sets us up to use Bundler's modern platform handling.
This threatened to make all of our workarounds for Bundler bugs in our packaging scripts even more heinous than they already were. Testing indicates that Bundler sometimes still does the wrong thing if some gems from the bundle are already installed. So how do we deal with that now? It's "easy"! We "just" run Bundler with our own GEM_HOME and GEM_PATH that only have Bundler available, so it can't find any other gems and therefore we can't trigger the bug.
- All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
- See above
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- If you are okay with this general approach, I will file issues for the TODOs noted in the comments.
- Code is tested and passing, both automated and manual, what manual testing was done is described.
- See above. Once this branch is ready to merge, I propose we test further by:
- Build a branch off 3.2-release with these changes.
- Build these packages as 3.2.0-2. I think it's fair to say these are "just" packaging changes, not changes to any Arvados service, so it's accurate to reflect that in the package iteration.
- Test the packages on pirca.
- On success, publish these packages to the release repository and merge to main.
- See above. Once this branch is ready to merge, I propose we test further by:
- Tested code incorporates recent main branch changes.
- Yes
- New or changed UI/UX and has gotten feedback from stakeholders.
- N/A
- Documentation has been updated.
- N/A I think. I have tried to make sure a comment explains why all the key implementation choices are here.
- Behaves appropriately at the intended scale (describe intended scale).
- No change
- Considered backwards and forwards compatibility issues between client and server.
- N/A
- Follows our coding standards and GUI style guidelines.
- N/A (no shell style guide)
Updated by
Updated by Brett Smith 4 months ago
· Edited
- Release set to 84
Brett Smith wrote in #note-8:
- Build a branch off 3.2-release with these changes.
- Build these packages as 3.2.0-2. I think it's fair to say these are "just" packaging changes, not changes to any Arvados service, so it's accurate to reflect that in the package iteration.
- Test the packages on pirca.
- On success, publish these packages to the release repository and merge to main.
There is a new branch 23000-gemfile-platform-bugfix that has all the same code but is based on 3.2-staging to prepare new release packages.
I built 3.2.0-2 and that revealed an oversight that our new GEM_HOME and GEM_PATH needed to be carried forward to the systemd unit which I'm inclined to consider an "obvious" bugfix. c05a5735ede34ebd7713b52b98368a0187fa13a7 got built as 3.2.0-3.
build-and-publish-rc-packages: #289
test-provision: #1361 - main is failing the same way right now, I need to investigate this but at least it shows the API server starting and running.
I installed 3.2.0-3 on pirca and it's running okay right now too.
I have asked the original reporter if they want to test 3.2.0-3 before we promote it to release.
Updated by Brett Smith 4 months ago
· Edited
- Status changed from In Progress to Resolved
Because this kind of packaging fix isn't common for us, I wasn't sure how to communicate it through the stable release channel. I decided to sort of take the middle ground and promote all 3.2.0-3 packages to stable for all of our RPM-based distributions, but not our DEB-based ones. publish-packages-to-stable-repo: #298
I think this helps communicate the scope of the fix while still letting administrators pin all of their Arvados packages on one specific version for their distribution if they like. If we change our mind later it's no big deal to promote the 3.2.0-3 packages for the DEB-based distributions.