Project

General

Profile

Actions

Story #16447

open

Improve container image handling

Added by Peter Amstutz over 4 years ago. Updated almost 2 years ago.

Status:
In Progress
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
11/01/2022
Due date:
05/31/2023 (about 19 months late)
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

We want to improve UX for common workflows like:
  • Use Workbench2 (and no shell node) to run a workflow that depends on docker images that are available on dockerhub
  • Build an image from a Dockerfile or git repo, and use that image to run Arvados containers without pushing it to dockerhub
  • Build an image and share it with other users on your cluster without pushing it to dockerhub
  • Share a project that contains a workflow execution plus all of the docker images needed to re-run it even after the referenced images on dockerhub/arvados have been updated or removed
Proposed features:
  • Improve tagging behavior: Avoid inconvenient/mysterious behaviors (e.g., image collection is visible by two users but only one can use it as an image because the tag link isn't shared) by using collection properties instead of tag links to identify docker images. #19846
  • Server-side pull: Workflow runner, when running without access to a local docker daemon (e.g., inside a container submitted by Workbench), can use a container request to pull an externally hosted (dockerhub) image and refer to that image by PDH when executing workflow steps. #11724
  • Server-side build: Workflow runner or other client without access to a local docker daemon (or preferring to leave a better provenance trail) can use a container request to build an image from a specified Dockerfile. #13794
  • Arvados-hosted image namespace: Each cluster has a "docker images" project. Any user with username X can use arv-keepdocker (or a new arvados-client command?) to save docker images in a subproject named X. A container request with container image "arvados:X/Y" will use the image saved in the collection named Y in the subproject named X, subject to usual permissions.
  • Implicit pull at runtime: A container request with container image "docker:abc/def" causes Arvados to fetch/update "abc/def" from dockerhub into an arvados collection, and use that collection PDH as the image in the resulting container.

See Build docker images as part of a workflow


Related issues 4 (3 open1 closed)

Related to Arvados - Story #11724: Pull Docker images without requiring Docker on clientResolvedTom Clegg12/06/2022

Actions
Related to Arvados - Story #13794: Build Docker images without requiring Docker on clientNew

Actions
Related to Arvados - Story #19846: Use collection properties instead of links to tag docker imagesNewTom Clegg01/18/2023

Actions
Related to Arvados - Feature #19860: Support "pull image" container requestNewTom Clegg

Actions
Actions #1

Updated by Peter Amstutz over 4 years ago

  • Due date set to 09/01/2021
  • Start date set to 06/01/2021
Actions #2

Updated by Peter Amstutz almost 4 years ago

  • Due date changed from 09/01/2021 to 06/30/2021
  • Start date changed from 06/01/2021 to 01/01/2021
Actions #3

Updated by Peter Amstutz almost 4 years ago

  • Status changed from New to In Progress
Actions #4

Updated by Peter Amstutz almost 4 years ago

  • Due date changed from 06/30/2021 to 12/31/2021
  • Start date changed from 01/01/2021 to 09/01/2021
Actions #5

Updated by Tom Clegg over 3 years ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz over 3 years ago

  • Due date changed from 12/31/2021 to 01/31/2022
  • Start date changed from 09/01/2021 to 10/01/2021
Actions #7

Updated by Peter Amstutz about 3 years ago

  • Due date changed from 01/31/2022 to 03/31/2022
  • Start date changed from 10/01/2021 to 01/01/2022
Actions #8

Updated by Peter Amstutz about 3 years ago

  • Due date changed from 03/31/2022 to 09/30/2022
  • Start date changed from 01/01/2022 to 06/01/2022
Actions #9

Updated by Peter Amstutz almost 3 years ago

  • Due date changed from 09/30/2022 to 11/30/2022
  • Start date changed from 06/01/2022 to 08/01/2022
Actions #10

Updated by Peter Amstutz over 2 years ago

  • Due date changed from 11/30/2022 to 12/31/2022
  • Start date changed from 08/01/2022 to 09/01/2022
Actions #11

Updated by Peter Amstutz over 2 years ago

  • Related to Story #11724: Pull Docker images without requiring Docker on client added
Actions #12

Updated by Peter Amstutz over 2 years ago

  • Related to Story #13794: Build Docker images without requiring Docker on client added
Actions #13

Updated by Peter Amstutz about 2 years ago

  • Start date changed from 09/01/2022 to 11/01/2022
Actions #14

Updated by Tom Clegg about 2 years ago

  • Related to Story #19846: Use collection properties instead of links to tag docker images added
Actions #15

Updated by Tom Clegg about 2 years ago

  • Description updated (diff)
Actions #16

Updated by Tom Clegg about 2 years ago

  • Related to Feature #19860: Support "pull image" container request added
Actions #17

Updated by Peter Amstutz almost 2 years ago

  • Due date changed from 12/31/2022 to 05/31/2023
Actions

Also available in: Atom PDF