Arvados

19070-update-workflow-deps @ bea50608be40f5ecf218fa5be75213579ad95eca

• add --copy-deps option, when enabled, dependencies of the workflow (input collections and docker images) are copied into the destination project
• copy-deps is default behavior for --create-workflow and --update-workflow
• when executing workflows from command line, does not copy anything by default (can still provide --copy-deps)
• added integration test which tries the various options and checks the results

developer-run-tests: #3086

19070-update-workflow-deps @ 1eb881796a672247db251fd7298b0a7253f83259

• Everything from above, but also make --match-submitter-images default behavior for --create-workflow and --update-workflow

developer-run-tests: #3096

19070-update-workflow-deps @ 8b4b7e7edb68a9c252bcc57b2d291b0f6e7a7ce9

• Fix tests

developer-run-tests: #3098

Some questions & comments:

• In file sdk/python/arvados/commands/keepdocker.py:L410 Is there an advantage of uploading the docker image as a first attempt instead of using one that's already in Arvados?
• Typo 'depenencies' at --no-copy-deps's help message
• In file sdk/cwl/arvados_cwl/executor.py: L547 refers to runtimeContext.update_workflow but L554 refers to self.runtimeContext.update_worflow, how both context differentiate between each other? the one passed to arv_executor() should have porecedence?
• This behavior change may need some documentation updates and probably a migration note, in case clients rely on the old way of doing things, wdyt?

Lucas Di Pentima wrote:

Some questions & comments:

• In file sdk/python/arvados/commands/keepdocker.py:L410 Is there an advantage of uploading the docker image as a first attempt instead of using one that's already in Arvados?

Because if the image exists in Docker locally, the assumption is that we want to match the image id of the local copy with the Arvados copy, which means we have to find out what the local version is.

As a fallback behavior, if you don't have the image or don't have Docker, you can still use arv-keepdocker to copy an image from one project to another.

• Typo 'depenencies' at --no-copy-deps's help message

Fixed.

• In file sdk/cwl/arvados_cwl/executor.py: L547 refers to runtimeContext.update_workflow but L554 refers to self.runtimeContext.update_worflow, how both context differentiate between each other? the one passed to arv_executor() should have porecedence?

The code was being sloppy, some places referred to the runtimeContext that was passed in, and other places referred to the runtimeContext from arvRunner (because it wasn't passing in runtimeContext). I cleaned this up, now it uses the passed-in runtimeContext almost everywhere. The only reason I didn't do it the first time is because it was extra work (but I went ahead with it since you called it out).

• This behavior change may need some documentation updates and probably a migration note, in case clients rely on the old way of doing things, wdyt?

I updated the documentation.

19070-update-workflow-deps @ 45e2ea82d68e74774bf61f6966e3777973e61845

developer-run-tests: #3118

This LGTM, thanks!