Project

General

Profile

Actions
Copy link

Bug #17962

closed

[arv-copy] blows up when copying a project with a registered workflow in it

Added by Ward Vandewege over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Peter Amstutz
Category:
-
Target version:
2021-11-24 sprint
Start date:
11/17/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release:
Arvados 2.3.1
Release relationship:
Auto

Description

Trying to copy a test project from pirca to tordo.

First, the project contained only collections and container requests. The collections were copied, container requests ignored (as expected).

# on shell.tordo
$ arv-copy pirca-j7d0g-bbu5p6jsjjfokta
pirca-4zz18-ccifrmbm41a3z84: 0M / 0M 100.0% 
pirca-4zz18-aneymw4b20pwug0: 0M / 0M 100.0% 
pirca-4zz18-smataa81t834uug: 0M / 0M 100.0% 
pirca-4zz18-k03bijojmx1m3n1: 0M / 0M 100.0% 
pirca-4zz18-ldybp10d3adj60u: 0M / 0M 100.0% 
pirca-4zz18-nuz3ntrreffkanr: 0M / 0M 100.0% 
pirca-4zz18-3xsu89n5mbiy2l6: 0M / 0M 100.0% 
pirca-4zz18-ghfqhzbn8z9bghc: 0M / 0M 100.0% 
pirca-4zz18-uvwqkky5kjtthem: 0M / 0M 100.0% 
2021-08-02 15:17:03 arvados.arv-copy[26659] INFO: 
2021-08-02 15:17:03 arvados.arv-copy[26659] INFO: Success: created copy with uuid tordo-j7d0g-2s0oya2vzqps25h

Then, I copied a registered workflow into the project, and tried to copy it again:

$ arv-copy pirca-j7d0g-bbu5p6jsjjfokta
Traceback (most recent call last):
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/reader.py", line 105, in stream
    return self._stream
AttributeError: 'SafeLoader' object has no attribute '_stream'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/arv-copy", line 7, in <module>
    main()
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/arvados/commands/arv_copy.py", line 144, in main
    result = copy_project(args.object_uuid, src_arv, dst_arv, args.project_uuid, args)
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/arvados/commands/arv_copy.py", line 683, in copy_project
    copy_workflow(w["uuid"], src, dst, args)
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/arvados/commands/arv_copy.py", line 291, in copy_workflow
    wf_def = yaml.safe_load(wf["definition"])
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/main.py", line 994, in safe_load
    return load(stream, SafeLoader, version)
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/main.py", line 947, in load
    loader = Loader(stream, version, preserve_quotes=preserve_quotes)
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/loader.py", line 40, in __init__
    Scanner.__init__(self, loader=self)
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/scanner.py", line 82, in __init__
    self.reset_scanner()
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/scanner.py", line 104, in reset_scanner
    self.fetch_stream_start()
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/scanner.py", line 446, in fetch_stream_start
    mark = self.reader.get_mark()
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/reader.py", line 180, in get_mark
    if self.stream is None:
  File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/ruamel/yaml/reader.py", line 107, in stream
    raise YAMLStreamError('input stream needs to specified')
ruamel.yaml.error.YAMLStreamError: input stream needs to specified

Interestingly, this doesn't seem to happen all the time:

ward@shell:~$ dpkg -l |grep arv
ii  arvados-client                         2.3.0~dev20210803154620-1                   amd64        Arvados command line tool (beta)
ii  arvados-src                            2.3.0~dev20210803154620-1                   all          The Arvados source code
ii  libpam-arvados-go                      2.3.0~dev20210803154620-1                   amd64        Arvados PAM authentication module
ii  python3-arvados-cwl-runner             2.3.0~dev20210729201354-1                   amd64        Arvados Common Workflow Language runner
ii  python3-arvados-fuse                   2.3.0~dev20210726143205-1                   amd64        Arvados FUSE driver
ii  python3-arvados-python-client          2.3.0~dev20210726143205-1                   amd64        Arvados client library

ward@shell:~$ arv-copy pirca-j7d0g-qv5r6tyawa0fri3 --project-uuid ce8i5-j7d0g-3b990qlglr7cmse
pirca-4zz18-3ik849l082fqjcz: 0M / 0M 100.0% 
pirca-4zz18-xa0i7qjide8td5d: 156M / 156M 100.0% 
pirca-4zz18-n9os5bjq4ipwe7d: 23M / 23M 100.0% 
pirca-4zz18-xos3zqns62comjb: 94M / 94M 100.0% 
pirca-4zz18-ckim3zvdmbf3i05: 0M / 0M 100.0% 
pirca-4zz18-3n2p2xu20q42it0: 259M / 259M 100.0% 
2021-08-03 17:26:39 arvados.arv-copy[28298] INFO: 
2021-08-03 17:26:39 arvados.arv-copy[28298] INFO: Success: created copy with uuid ce8i5-j7d0g-fqjqh6cs9nezwey

Subtasks 1 (0 open — 1 closed)

Task #18038: Review 17962-check-wf-definitionResolvedPeter Amstutz11/17/2021

Actions
Actions
Copy link
 #1

Updated by Ward Vandewege over 3 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Peter Amstutz over 3 years ago

That's really weird. Seems like some kind of ruamel.yaml failure. Bad version maybe?

Actions
Copy link
 #3

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-08-18 sprint to 2021-09-01 sprint
Actions
Copy link
 #4

Updated by Ward Vandewege over 3 years ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by Ward Vandewege over 3 years ago

  • Description updated (diff)
Actions
Copy link
 #6

Updated by Peter Amstutz over 3 years ago

  • Assigned To set to Peter Amstutz
Actions
Copy link
 #7

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-09-01 sprint to 2021-09-15 sprint
Actions
Copy link
 #8

Updated by Peter Amstutz over 3 years ago

  • Assigned To deleted (Peter Amstutz)
Actions
Copy link
 #9

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-09-15 sprint to 2021-09-29 sprint
Actions
Copy link
 #10

Updated by Peter Amstutz over 3 years ago

  • Assigned To set to Peter Amstutz
Actions
Copy link
 #11

Updated by Peter Amstutz about 3 years ago

  • Target version changed from 2021-09-29 sprint to 2021-10-13 sprint
Actions
Copy link
 #12

Updated by Peter Amstutz about 3 years ago

  • Target version changed from 2021-10-13 sprint to 2021-10-27 sprint
Actions
Copy link
 #13

Updated by Peter Amstutz about 3 years ago

  • Target version changed from 2021-10-27 sprint to 2021-11-10 sprint
Actions
Copy link
 #14

Updated by Peter Amstutz about 3 years ago

  • Target version changed from 2021-11-10 sprint to 2021-11-24 sprint
Actions
Copy link
 #15

Updated by Peter Amstutz about 3 years ago

  • Release set to 42
Actions
Copy link
 #16

Updated by Peter Amstutz about 3 years ago

  • Release deleted (42)
Actions
Copy link
 #17

Updated by Peter Amstutz about 3 years ago

17962-check-wf-definition @ 972272c42f7ad8daaef29d3d981968e566f3f087

developer-run-tests: #2806

The reason this failed was because the specific workflow record in the bug report had null in the "definition" fiend. I don't know how that happened, but arv-copy didn't like it.

I've improved the error handling for arv-copy a little bit.

$ arv-copy pirca-j7d0g-bbu5p6jsjjfokta
2021-11-17 16:15:33 arvados.arv-copy[1853791] WARNING: Workflow object pirca-7fd4e-hx0a8x0sskhd1od has an empty or null definition, it won't do anything.
2021-11-17 16:15:33 arvados.arv-copy[1853791] WARNING: Warning: created copy with uuid x2z00-j7d0g-hqfaopz1teptkn0 but failed to copy some items: 
Error while copying pirca-7fd4e-hx0a8x0sskhd1od: input stream needs to specified
Actions
Copy link
 #18

Updated by Peter Amstutz about 3 years ago

  • Release set to 45
Actions
Copy link
 #19

Updated by Lucas Di Pentima about 3 years ago

  • At line 164: I think that print() call was for debugging?

LGTM otherwise, thanks!

Actions
Copy link
 #20

Updated by Peter Amstutz about 3 years ago

Lucas Di Pentima wrote:

  • At line 164: I think that print() call was for debugging?

LGTM otherwise, thanks!

Actually it was on purpose, it makes it usable in a script (you can capture stdout to get the uuid of the copy). Although it's technically out of scope of the ticket so I can pull it out if you think I should.

Actions
Copy link
 #21

Updated by Lucas Di Pentima about 3 years ago

I think it's a good idea but also guess that it should be validated with a test and maybe some documentation (ie: on the command's --help message)

Actions
Copy link
 #22

Updated by Peter Amstutz about 3 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF