Project

General

Profile

Actions

Feature #16583

open

Can programmatically distinguish between final outputs (results of top-level containers) and intermediate outputs.

Added by Peter Amstutz almost 4 years ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
Workbench2
Target version:
Start date:
05/19/2022
Due date:
% Done:

33%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Output collections have properties "container_request: <uuid>" and "type: output". This allows workbench2 to offer a filter on outputs, logs, and general.

Similar to how I can filter on just top level containers, I'd like to be able to filter on top level outputs. We also want to be able to do scripted cleaning up of intermediate outputs while leaving the final output alone.

Top level containers have "requested_by_container_uuid: null". Technically we could look up the container request for each collection to determine if it is top level or not, but that can't be written as a single query filter.

Proposed approach:

  1. Add ability to set properties of container outputs: #17004
  2. Arvados-cwl-runner starts marking intermediate and final collections, using output_properties to set it
  3. Add a rake task that the admin can optionally use to infer the intermediate and final collection output property from requested_by_container_uuid.
  4. Add to Workbench2 the ability to filter out intermediate collections from project display.
  5. Document the rake task in the upgrade notes

Subtasks 3 (2 open1 closed)

Task #19118: Review 16583-intermediate-collections on Workbench 2ResolvedPeter Amstutz05/19/2022

Actions
Task #19163: Write rake taskNewPeter Amstutz05/19/2022

Actions
Task #19173: Review rake task branchNewLucas Di Pentima05/19/2022

Actions

Related issues

Related to Arvados Epics - Story #17848: CWL runner improvementsIn Progress07/01/202106/30/2023

Actions
Blocked by Arvados - Feature #17004: Can set output_properties on output collection of a container requestResolvedPeter Amstutz05/13/2022

Actions
Actions #1

Updated by Peter Amstutz almost 4 years ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz almost 4 years ago

  • Target version changed from 2020-08-12 Sprint to 2020-08-26 Sprint
Actions #3

Updated by Peter Amstutz almost 4 years ago

  • Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint
Actions #4

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint
Actions #5

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-09-23 Sprint to 2020-10-07 Sprint
Actions #6

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-10-07 Sprint to 2020-10-21 Sprint
Actions #7

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-10-21 Sprint to 2020-11-04 Sprint
Actions #8

Updated by Peter Amstutz over 3 years ago

  • Description updated (diff)
Actions #9

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-11-04 Sprint to 2020-11-18
Actions #10

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-11-18 to 2020-11-04 Sprint
Actions #11

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-11-04 Sprint to 2020-11-18
Actions #12

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-11-18 to 2020-12-02 Sprint
Actions #13

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-12-02 Sprint to 2020-12-16 Sprint
Actions #14

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2020-12-16 Sprint to 2021-01-06 Sprint
Actions #15

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-01-06 Sprint to 2021-01-20 Sprint
Actions #16

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-01-20 Sprint to 2021-02-03 Sprint
Actions #17

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-02-03 Sprint to 2021-02-17 sprint
Actions #18

Updated by Peter Amstutz over 3 years ago

  • Target version deleted (2021-02-17 sprint)
Actions #19

Updated by Peter Amstutz about 3 years ago

  • Related to Story #16011: CWL support, docs, training, website added
Actions #20

Updated by Peter Amstutz about 3 years ago

  • Assigned To set to Jiayong Li
Actions #21

Updated by Peter Amstutz about 3 years ago

  • Assigned To deleted (Jiayong Li)
Actions #22

Updated by Peter Amstutz almost 3 years ago

Actions #23

Updated by Peter Amstutz almost 3 years ago

  • Related to deleted (Story #16011: CWL support, docs, training, website)
Actions #24

Updated by Peter Amstutz almost 3 years ago

  • Related to Feature #17004: Can set output_properties on output collection of a container request added
Actions #25

Updated by Peter Amstutz almost 3 years ago

  • Description updated (diff)
Actions #26

Updated by Peter Amstutz almost 3 years ago

  • Related to deleted (Feature #17004: Can set output_properties on output collection of a container request)
Actions #27

Updated by Peter Amstutz almost 3 years ago

  • Blocked by Feature #17004: Can set output_properties on output collection of a container request added
Actions #28

Updated by Peter Amstutz about 2 years ago

  • Target version set to 2022-04-27 Sprint
Actions #29

Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
Actions #30

Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Actions #31

Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2022-05-25 sprint to 2022-06-08 sprint
Actions #32

Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2022-06-08 sprint to 2022-05-25 sprint
Actions #33

Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)
Actions #34

Updated by Peter Amstutz about 2 years ago

  • Assigned To set to Peter Amstutz
Actions #35

Updated by Peter Amstutz about 2 years ago

  • Category changed from Crunch to Workbench2
Actions #37

Updated by Peter Amstutz almost 2 years ago

  • Status changed from New to In Progress
Actions #38

Updated by Lucas Di Pentima almost 2 years ago

  • There's a unit test failing.
  • I think it would be convenient to add a unit test for this new filter type at src/store/resource-type-filters/resource-type-filters.test.ts
Actions #39

Updated by Peter Amstutz almost 2 years ago

Lucas Di Pentima wrote:

  • There's a unit test failing.
  • I think it would be convenient to add a unit test for this new filter type at src/store/resource-type-filters/resource-type-filters.test.ts

16583-intermediate-collections @ arvados-workbench2|015c7956121aceeade3d92243022c51e552d7a32

  • Fix test & added a test

developer-tests-workbench2: #696

Actions #40

Updated by Peter Amstutz almost 2 years ago

  • Release set to 51
Actions #42

Updated by Lucas Di Pentima almost 2 years ago

Test re-re-run: developer-tests-workbench2: #700 (the previous was cancelled after 45 minutes)

Actions #44

Updated by Lucas Di Pentima almost 2 years ago

This LGTM, thanks.

Actions #45

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-05-25 sprint to 2022-06-08 sprint
Actions #46

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-06-08 sprint to 2022-06-22 Sprint
Actions #47

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-06-22 Sprint to 2022-07-06
Actions #48

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-07-06 to 2022-07-20
Actions #49

Updated by Peter Amstutz almost 2 years ago

  • Release changed from 51 to 52
Actions #50

Updated by Peter Amstutz almost 2 years ago

  • Target version deleted (2022-07-20)
Actions #51

Updated by Peter Amstutz almost 2 years ago

  • Release changed from 52 to 51
Actions #52

Updated by Peter Amstutz over 1 year ago

  • Target version set to 2023-01-18 sprint
Actions #53

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2023-01-18 sprint to To be groomed
Actions #54

Updated by Peter Amstutz over 1 year ago

  • Status changed from In Progress to New
Actions

Also available in: Atom PDF