Feature #23380
closed
Bake the latest Go toolchain in each new Jenkins image (in addition to the one currently in go.mod)
Description
Currently:
When building a Jenkins image we install the Go toolchain specified by go.mod. When we update go.mod, each Jenkins worker has to re-download the new version until we build a new Jenkins image. This is somewhat wasteful, and implicitly turns every toolchain update into an additional ops task.
Proposed improvement:
At any moment when we're building a Jenkins image, the latest Go release is a pretty good guess of the toolchain we're most likely to update to next. If we preload that into the image, we will have the option to update to that release without rebuilding an image or doing repetitive downloads.
Auto-downloaded toolchains are stored in GOPATH.
Preloading the latest Go release looks like this (presumably after running run-tests.sh --only nothing to create ~/jenkins/tmp/GOPATH):
latest="$(curl -Ls 'https://go.dev/dl/?mode=json' | jq -r '.[0].version')" GOPATH=~jenkins/tmp/GOPATH GOTOOLCHAIN=$latest go version
Updated by Tom Clegg 3 months ago
23380-show-toolchains @ a4a997f0bfd2eb2357c483bb17015692173d0887
Shows additional toolchains already available in tempdir GOPATH, if any. This way we can look at a recent test run to find out whether updating go.mod to toolchain X would trigger extra downloads / building a new Jenkins image.
Updated by Brett Smith 3 months ago
So, like, the implementation is straightforward (thanks) and I don't really object to adding it, but I'm worried it's not going to be as big of a win as you/we hope.
Tom Clegg wrote:
When building a Jenkins image we install the Go toolchain specified by go.mod. When we update go.mod, each Jenkins worker has to re-download the new version until we build a new Jenkins image. This is somewhat wasteful, and implicitly turns every toolchain update into an additional ops task.
So, I will say in general, in my view building a new Jenkins image shouldn't be a big deal, and it shouldn't be a problem for it to be basically routine. When we do it, we also get Debian security updates, and we refresh everything in the test temp directory (Ruby gems, Python wheels, etc.), not just the Go toolchain. So it's nice do periodically even when the original motivation is a "small" change like a new Go toolchain.
The last few builds have been reliable and in my experience take ~30 minutes of ops time: ~10 to plan and start the build, ~10 to reconfigure Jenkins to use it, and ~10 to review results. Yeah I wouldn't want to do it every week but ~once a month seems reasonable to me, and at least right now I don't see us updating Go more often than that.
At any moment when we're building a Jenkins image, the latest Go release is a pretty good guess of the toolchain we're most likely to update to next.
I agree it's the best guess we can make but I wonder how often it'll end up being accurate. It seems like the best use of our time to plan to upgrade to the latest point release of the chosen version, and that might be newer than whatever was available when we built the last image. When that happens, we get no benefit, we end up in the same situation we have now.