Arvados

Pushed branch 23040-ansible-set-up-ubuntu-debian-repo-sources; see diff .

In fact, I don't know why we re-create the sources list files for the default Ubuntu/Debian and their -security repos, as opposed to leaving them alone. Could you help me by explaining this? Thanks.

Zoë Ma wrote in #note-2:

In fact, I don't know why we re-create the sources list files for the default Ubuntu/Debian and their -security repos, as opposed to leaving them alone. Could you help me by explaining this? Thanks.

We need to make sure contrib (on Debian)/universe (on Ubuntu) is enabled so we can find our dependencies. At least with Debian server installs, it's not uncommon for them to have only the main suite enabled.

I see, thank you.

On Debian 12 and (I think) Ubuntu 22.04, APT uses the (legacy?) single-line source format for its default package and security repos at /etc/apt/sources.list. As a result, after Arvados installation, running apt update creates many lines of warning that looks like

W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:11 and /etc/apt/sources.list.d/debian.sources:1

which I guess is harmless but annoying. I am thinking of adding a step that moves the /etc/apt/sources.list file to a backup path. But I need to know if what I'm doing here is in the right direction. Thanks.

Zoë Ma wrote in #note-4:

On Debian 12 and (I think) Ubuntu 22.04, APT uses the (legacy?) single-line source format for its default package and security repos at /etc/apt/sources.list. As a result, after Arvados installation, running apt update creates many lines of warning… which I guess is harmless but annoying. I am thinking of adding a step that moves the /etc/apt/sources.list file to a backup path.

I don't think this is a good idea because other tools might've added other repositories to that file. Even on Ubuntu installs, even if it just has the multiverse component enabled, turning that off could disrupt the system. A warning is annoying but also the least disruptive option. An administrator who knows it's safe to delete on their own systems can do so.

Overall this branch looks fine and I'd be happy to merge it, but can you please explain if there's any particular reason you enabled the backports component on Ubuntu? Thanks.

but can you please explain if there's any particular reason you enabled the backports component on Ubuntu? Thanks.

There's no other reason I did that, except that the backports component is enabled by default on Ubuntu 24.04 (and 22.04 too, in the old-style /etc/apt/sources.list file).

Zoë Ma wrote in #note-8:

There's no other reason I did that, except that the backports component is enabled by default on Ubuntu 24.04 (and 22.04 too, in the old-style /etc/apt/sources.list file).

This makes sense, but now that you mention it, I'm realizing we have the same problem with debian.sources or ubuntu.sources that we have with sources.list: the administrator might've customized it and we shouldn't overwrite those customizations. I think this wasn't on my radar when I wrote the initial code because Debian was still based on sources.list, but with the recent Debian 13 release we'll have the same problem there that we have here.

I'm open to a second opinion from Lucas but I think what we should do is instead of overwriting files, make new files like arvados-distro.sources and put only the sources we need in there. That way we minimize our impact on the system while providing as much as possible to help ensure a successful installation. This means that we'll get the warning on every distribution, not just some of them, but as far as I'm concerned that's an apt problem. Most programs let you redefine/override configuration without so much trouble. And again, a warning is the least bad problem compared to an unsuccessful install or disrupting the existing system configuration.

Then we don't need any changes to the suites: Arvados doesn't rely on backports so doesn't need to set them up. We should keep your changes to define the Signed-By parameter, though, that's a great improvement.

Thank you for the suggestion.

I'm open to a second opinion from Lucas but I think what we should do is instead of overwriting files, make new files like arvados-distro.sources and put only the sources we need in there.

I just did this in the following diffs . The repo files added by Ansible would be renamed arvados-distro[-security].sources and will not overwrite any system defaults. Tested this on Debian 12 and Ubuntu 24.04, and both worked (except the 'annoying' but harmless duplicate source warning messages from APT)

Brett Smith wrote in #note-9:

I'm open to a second opinion from Lucas but I think what we should do is instead of overwriting files, make new files like arvados-distro.sources and put only the sources we need in there. That way we minimize our impact on the system while providing as much as possible to help ensure a successful installation. This means that we'll get the warning on every distribution, not just some of them, but as far as I'm concerned that's an apt problem. Most programs let you redefine/override configuration without so much trouble. And again, a warning is the least bad problem compared to an unsuccessful install or disrupting the existing system configuration.

I agree with this. The salt installer does the same thing, it installs its own arvados.list file instead of touching the default distro's sources.list.

One thing I'm would like to point out is that I'm not sure using distro-dependant sources file naming (eg: arvados-jammy.sources) would be convenient for those cases when the distro OS is upgraded to a new release, then the old release file would be left lingering and causing problems that would require manual intervention. If by arvados-distro.sources you meant literally that name, then disregard this comment :)

Lucas Di Pentima wrote in #note-11:

If by arvados-distro.sources you meant literally that name, then disregard this comment :)

I did mean literally that name, and that's what Zoë wrote, so we're good.

I rebased the branch on main to remove the reverted commit churn. I also changed the field name in defaults from gpg_key to signed_by: all the current field names match the parameters of the Ansible task, so this keeps things consistent. But with that, this is merged, thanks!