Arvados

21362-rhel9-packages @ 7aadd65e4adba4b9c386f4553f41acc205247be8 - build-packages-rocky9: #1

• All agreed upon points are implemented / addressed.
  • Implements everything in the Arvados repository plus an initial Jenkins job
  • Uses lessons learned to slim down package test Dockerfiles to make testing more effective
  • Removes some unsupported distribution cruft
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • This branch does not resolve the ticket, there's follow-up infrastructure work that needs to happen, but it practically depends on the branch getting merged first.
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • See above
  • Also tested a slimmed-down Debian test image on my development system by running:

    WORKSPACE="$PWD" build/run-build-test-packages-one-target.sh --target debian12 --arch amd64 --force-build

    This succeeded.
  • During my local testing of rocky9 I occasionally encountered this error when trying to install packages:

    (microdnf:13): librepo-WARNING **: 18:00:03.250: LRO_MIRRORLISTURL processing failed: No URLs in mirrorlist
    error: cannot update repo 'devel': Cannot prepare internal mirrorlist: No URLs in mirrorlist
    

    This is a pure Rocky infrastructure problem. You can search this error and find various reports of it happening in the past and being addressed by them. We can work around it if it becomes persistent (by hardcoding repository URLs), but it didn't occur once on the Jenkins test run, and I would rather not add workaround code we don't need.
• Documentation has been updated.
  • Yes
• Behaves appropriately at the intended scale (describe intended scale).
  • No change
• Considered backwards and forwards compatibility issues between client and server.
  • N/A
• Follows our coding standards and GUI style guidelines.
  • N/A (no style guide for shell or Dockerfiles)

The GPG key we currently use to sign packages is so old, it is not considered secure by RHEL9. See #22603.

IMO we should generate a new key with modern algorithms, then use it to sign packages for RHEL9 as well as any new distributions going forward like Debian 13. This means we'll need to teach all our package publication infrastructure to parameterize the signing key by distribution. But that's probably less work for us than trying to switch the signing key everywhere at once, which would require intervention on every deployed cluster to keep using the Arvados package repository.

21362-rhel9-packages LGTM, thanks.

21362-rhel9-packages will be merged into 3.1.2 because it includes generally useful API server bugfixes. It will also add RHEL 9 packaging code but that'll be a noop until we start running Jenkins jobs for it.

Brett Smith wrote in #note-15:

IMO we should generate a new key with modern algorithms, then use it to sign packages for RHEL9 as well as any new distributions going forward like Debian 13. This means we'll need to teach all our package publication infrastructure to parameterize the signing key by distribution. But that's probably less work for us than trying to switch the signing key everywhere at once, which would require intervention on every deployed cluster to keep using the Arvados package repository.

Agreed at engineering meeting that this is the plan.

Brett Smith wrote in #note-15:

This means we'll need to teach all our package publication infrastructure to parameterize the signing key by distribution.

In aptly this is the -gpg-key option to the aptly publish repo and aptly publish update commands. We'll need to update various ops scripts to start using this.

RPMs are signed by rpmsign which only respects ~/.rpmmacros which I'm not sure if it can be parameterized. However, that might be okay because unlike apt, yum knows the source of the GPG keys. We may be able to publish a new key and start using it right away, even for RHEL 8, and existing installs (that have gpgkey set as documented) will continue to work without interruption. This SO comment suggests that would work.

Brett Smith wrote in #note-23:

RPMs are signed by rpmsign which only respects ~/.rpmmacros which I'm not sure if it can be parameterized.

Nevermind, you can say rpmsign --define "_gpg_name KEY_ID". --define is a general RPM option that I didn't realize worked here.

curii-ops branch 21362-parametrize-gpg-key extends all of our package publication code to specify the GPG key used for signing. Even though there is no new key yet, what I'd like to do is get this polished, merged, deployed, and then see it work for some basic tasks. Once that's done, then we can create a new key and use it to add Rocky 9 to the package publication infrastructure. But it's helpful to check that none of the existing distros break first.

I did a bunch of general code cleanup as part of this. I have tried to organize the commits to make it easier to follow. I hope it's helpful.

I haven't really done any testing besides syntax checks because that's hard to do outside the infrastructure. I have probably added small bugs. I'm committed to dealing with those as they come up. I think you're just singing off on this general approach.

Brett Smith wrote in #note-25:

I did a bunch of general code cleanup as part of this. I have tried to organize the commits to make it easier to follow. I hope it's helpful.

It is, thanks for making the review easy.

I haven't really done any testing besides syntax checks because that's hard to do outside the infrastructure. I have probably added small bugs. I'm committed to dealing with those as they come up. I think you're just singing off on this general approach.

I think it's great, one improvement that might be worth doing is DRYing the GPG Key map in commit 2f93e09076161be144eaf34874939884fc994f0a, but if it makes things more complicated, it's OK the way it is now.

Other than that, LGTM. Thanks!

Lucas Di Pentima wrote in #note-28:

I think it's great, one improvement that might be worth doing is DRYing the GPG Key map in commit 2f93e09076161be144eaf34874939884fc994f0a, but if it makes things more complicated, it's OK the way it is now.

I thought about this but note they're not technically the same map. One uses keys with distribution codenames; the other uses our own distro+version keys like debian12 and ubuntu2404. With this difference there isn't too much to DRY up other than the one line declaring the key ID itself.

build-packages-multijob: #4717

deploy-to-tordo: #1796

With these successes we have some confidence that the new signing code works. We should be good to go ahead and add a new key for singing Rocky 9 packages.

curii-ops branch 21362-rocky9-packaging adds the new key, adds rocky9 support, and fixes one bug bug in the key parametrization code.

arvados branch 21362-rocky9-packaging updates the documentation so RHEL repositories use a different key URL by version. All the old URLs still work, and all existing releases have been updated so the new URLs also work for them, not just RHEL 9.

build-packages-rocky9: #28

• All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
  • Yes
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • After these branches are merged, all that's left to do is add build-packages-rocky9 to build-packages-multijob.
• Code is tested and passing, both automated and manual, what manual testing was done is described.
  • Now that initial rocky9 packages are uploaded, I can follow the new documentation (replacing /os/ with /dev/ in the yum.repos.d file to get development packges) and install a development version of arvados-client with no extra fuss.
• New or changed UI/UX and has gotten feedback from stakeholders.
  • N/A? I guess the new GPG key URL is a change but it seems straightforward, and it's a set-it-and-forget-it thing for sysadmins.
• Documentation has been updated.
  • Yes
• Behaves appropriately at the intended scale (describe intended scale).
  • No change
• Considered backwards and forwards compatibility issues between client and server.
  • Purely additional support
• Follows our coding standards and GUI style guidelines.
  • N/A (no relevant standards)

Updates LGTM -- There's one thing that might be useful, though: For users that are already using Arvados with Rocky8, a heads up note for the new GPG key URL wherever makes more sense (future release notes, or upgrade notes) so that they can update their preexisting package sources configs.

Lucas Di Pentima wrote in #note-34:

Updates LGTM -- There's one thing that might be useful, though: For users that are already using Arvados with Rocky8, a heads up note for the new GPG key URL wherever makes more sense (future release notes, or upgrade notes) so that they can update their preexisting package sources configs.

Pushed at ab7fa087ee171f1ccadfc017db8e0ac562d799bc, got sign-off in chat. Thanks.

Still need to add build-packages-rocky9 to the multijob but I'm going to hold off while I'm messing with #23000.