Arvados

21389-update-docker-jobs @ a4864d8ba31ace660e06b06d78afc4b9a83bacad

This branch makes it so that all our Python virtualenv Docker images are built using the same code. By eliminating the dependency on published packages, it makes future development and testing easier. It goes ahead and upgrades to Debian 12 along the way. See the commit message for more details.

• All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
  • And then some.
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • After this is merged, the corresponding Jenkins jobs will need updating. I expect it to be a very straightforward update.
• Code is tested and passing, both automated and manual, what manual testing was done is described.
  • I have manually tested building both images, with both upload options, without credentials. Builds are successful. Uploads run the right command, then fail for lack of credentials. developer-run-tests will not exercise this code.
• New or changed UI/UX and has gotten feedback from stakeholders.
  • Strictly developer-facing. I maybe should've gotten feedback about the new script, but well, it's not like the old scripts got feedback ahead of time. At least this uses a better argument parser and requires fewer options.
• Documentation has been updated.
  • Yes
• Behaves appropriately at the intended scale (describe intended scale).
  • Yes. The size of arvados/jobs increases from ~125MB to ~440MB. Most of this comes from the upgrade to Debian 12: I built an image with the old code, just s/bullseye/bookworm/g everywhere, and got a very similar size to the wheel-based image. Therefore I think if we want to optimize size that needs to be a separate ticket.
• Considered backwards and forwards compatibility issues between client and server.
  • I considered making the existing shell scripts just wrappers around the new script, but given how little they seem to be used it didn't seem worth it. There are no callers in the codebase itself, only Jenkins. The Jenkins builds can just call build_docker_image.py directly to make future maintenance easier.
• Follows our coding standards and GUI style guidelines.
  • Yes