Project

General

Profile

Actions
Copy link

Bug #12347

closed

[CWL] reuse hints appear to be broken

Added by Ward Vandewege about 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tom Clegg
Category:
-
Target version:
2017-10-11 Sprint
Start date:
09/28/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Using

sdk/cwl/tests/noreuse.cwl

from the arvados tree, I've tested on 9tee4 (head, crunchv2), tb05z (0.1.20170925193954.68bdf4c, crunchv1), and qr1hi (0.1.20170817171636.0b48006, crunchv1).

wardtest@shell.9tee4:~$ cwl-runner --debug noreuse.cwl 
2017-09-27 14:51:11 cwltool INFO: /usr/bin/cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 14:51:11 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 14:51:13 cwltool DEBUG: Parsed job order from command line: {
    "id": "noreuse.cwl" 
}
2017-09-27 14:51:13 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506523873.98
2017-09-27 14:51:13 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506523873.98 0.000186204910278
2017-09-27 14:51:13 arvados.cwl-runner.metrics DEBUG: ENTER run 1506523873.98
2017-09-27 14:51:14 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-ye3hso3hnsw5g10
2017-09-27 14:51:14 arvados.keep DEBUG: {u'9tee4-bi6l4-uwgvqzzw4z97qca': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-uwgvqzzw4z97qca'), (u'kind', u'arvados#keepService'), (u'etag', u'c65xiys117q01qz9s7zvjt66i'), (u'uuid', u'9tee4-bi6l4-uwgvqzzw4z97qca'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:19:29.655913000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:21.044273000Z'), (u'service_host', u'keep0.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep0.9tee4.arvadosapi.com:25107/')]), u'9tee4-bi6l4-1fum8ggyzc88g87': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-1fum8ggyzc88g87'), (u'kind', u'arvados#keepService'), (u'etag', u'dn6g7rkwx5gpwq0h053wyquax'), (u'uuid', u'9tee4-bi6l4-1fum8ggyzc88g87'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:20:27.578073000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:43.978890000Z'), (u'service_host', u'keep1.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep1.9tee4.arvadosapi.com:25107/')])}
2017-09-27 14:51:14 arvados.keep DEBUG: 99914b932bd37a50b983c5e7c90ae93b+2+A61f031b2c8357576abfe330f1dfb9632e618f99d@59de2fe2: ['http://keep1.9tee4.arvadosapi.com:25107/', 'http://keep0.9tee4.arvadosapi.com:25107/']
2017-09-27 14:51:14 arvados.keep DEBUG: Request: GET http://keep1.9tee4.arvadosapi.com:25107/99914b932bd37a50b983c5e7c90ae93b+2+A61f031b2c8357576abfe330f1dfb9632e618f99d@59de2fe2
2017-09-27 14:51:14 arvados.keep INFO: GET 200: 2 bytes in 88.7131690979 msec (0.000 MiB/sec)
2017-09-27 14:51:14 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 14:51:14 arvados.cwl-runner.metrics DEBUG: EXIT run 1506523874.84 0.865196943283
2017-09-27 14:51:14 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506523874.84
2017-09-27 14:51:14 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506523874.84 6.19888305664e-05
2017-09-27 14:51:14 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 14:51:14 cwltool INFO: Final process status is success

wardtest@shell.9tee4:~$ cwl-runner --debug noreuse.cwl 
2017-09-27 14:52:38 cwltool INFO: /usr/bin/cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 14:52:38 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 14:52:39 cwltool DEBUG: Parsed job order from command line: {
    "id": "noreuse.cwl" 
}
2017-09-27 14:52:40 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506523960.51
2017-09-27 14:52:40 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506523960.51 0.000166893005371
2017-09-27 14:52:40 arvados.cwl-runner.metrics DEBUG: ENTER run 1506523960.51
2017-09-27 14:52:40 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-y7f9m3nzjdldoso
2017-09-27 14:52:41 arvados.keep DEBUG: {u'9tee4-bi6l4-uwgvqzzw4z97qca': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-uwgvqzzw4z97qca'), (u'kind', u'arvados#keepService'), (u'etag', u'c65xiys117q01qz9s7zvjt66i'), (u'uuid', u'9tee4-bi6l4-uwgvqzzw4z97qca'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:19:29.655913000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:21.044273000Z'), (u'service_host', u'keep0.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep0.9tee4.arvadosapi.com:25107/')]), u'9tee4-bi6l4-1fum8ggyzc88g87': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-1fum8ggyzc88g87'), (u'kind', u'arvados#keepService'), (u'etag', u'dn6g7rkwx5gpwq0h053wyquax'), (u'uuid', u'9tee4-bi6l4-1fum8ggyzc88g87'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:20:27.578073000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:43.978890000Z'), (u'service_host', u'keep1.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep1.9tee4.arvadosapi.com:25107/')])}
2017-09-27 14:52:41 arvados.keep DEBUG: 99914b932bd37a50b983c5e7c90ae93b+2+A5804900581ce783388a56366a5d2908f238b03c4@59de3039: ['http://keep1.9tee4.arvadosapi.com:25107/', 'http://keep0.9tee4.arvadosapi.com:25107/']
2017-09-27 14:52:41 arvados.keep DEBUG: Request: GET http://keep1.9tee4.arvadosapi.com:25107/99914b932bd37a50b983c5e7c90ae93b+2+A5804900581ce783388a56366a5d2908f238b03c4@59de3039
2017-09-27 14:52:41 arvados.keep INFO: GET 200: 2 bytes in 25.9730815887 msec (0.000 MiB/sec)
2017-09-27 14:52:41 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 14:52:41 arvados.cwl-runner.metrics DEBUG: EXIT run 1506523961.53 1.02316498756
2017-09-27 14:52:41 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506523961.53
2017-09-27 14:52:41 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506523961.53 7.60555267334e-05
2017-09-27 14:52:41 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 14:52:41 cwltool INFO: Final process status is success

Even adding --disable-reuse doesn't help:

wardtest@shell.9tee4:~$ cwl-runner --disable-reuse --debug noreuse.cwl 
2017-09-27 15:16:13 cwltool INFO: /usr/bin/cwl-runner 58c6f3aa42f4f30fc4a764ca56ab1a198754b69b 1.0.20170925131954, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170828135420
2017-09-27 15:16:13 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///home/wardtest/noreuse.cwl'
2017-09-27 15:16:15 cwltool DEBUG: Parsed job order from command line: {
    "id": "noreuse.cwl" 
}
2017-09-27 15:16:15 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506525375.98
2017-09-27 15:16:15 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506525375.98 0.000164031982422
2017-09-27 15:16:15 arvados.cwl-runner.metrics DEBUG: ENTER run 1506525375.98
2017-09-27 15:16:16 arvados.cwl-runner INFO: [container noreuse.cwl] submitted container 9tee4-xvhdp-ktk14izbiujbnkc
2017-09-27 15:16:16 arvados.keep DEBUG: {u'9tee4-bi6l4-uwgvqzzw4z97qca': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-uwgvqzzw4z97qca'), (u'kind', u'arvados#keepService'), (u'etag', u'c65xiys117q01qz9s7zvjt66i'), (u'uuid', u'9tee4-bi6l4-uwgvqzzw4z97qca'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:19:29.655913000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:21.044273000Z'), (u'service_host', u'keep0.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep0.9tee4.arvadosapi.com:25107/')]), u'9tee4-bi6l4-1fum8ggyzc88g87': OrderedDict([(u'href', u'/keep_services/9tee4-bi6l4-1fum8ggyzc88g87'), (u'kind', u'arvados#keepService'), (u'etag', u'dn6g7rkwx5gpwq0h053wyquax'), (u'uuid', u'9tee4-bi6l4-1fum8ggyzc88g87'), (u'owner_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'created_at', u'2014-08-27T02:20:27.578073000Z'), (u'modified_by_client_uuid', u'9tee4-ozdt8-yft7aixdt2dx5ni'), (u'modified_by_user_uuid', u'9tee4-tpzed-ruo0j78b8s7jk57'), (u'modified_at', u'2014-08-27T02:20:43.978890000Z'), (u'service_host', u'keep1.9tee4.arvadosapi.com'), (u'service_port', 25107), (u'service_ssl_flag', False), (u'service_type', u'disk'), (u'read_only', False), ('_service_root', 'http://keep1.9tee4.arvadosapi.com:25107/')])}
2017-09-27 15:16:16 arvados.keep DEBUG: 99914b932bd37a50b983c5e7c90ae93b+2+Ab6bdb8008dc52ee665f3e33681a4a24d15c632df@59de35c0: ['http://keep1.9tee4.arvadosapi.com:25107/', 'http://keep0.9tee4.arvadosapi.com:25107/']
2017-09-27 15:16:16 arvados.keep DEBUG: Request: GET http://keep1.9tee4.arvadosapi.com:25107/99914b932bd37a50b983c5e7c90ae93b+2+Ab6bdb8008dc52ee665f3e33681a4a24d15c632df@59de35c0
2017-09-27 15:16:16 arvados.keep INFO: GET 200: 2 bytes in 25.8030891418 msec (0.000 MiB/sec)
2017-09-27 15:16:16 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 15:16:16 arvados.cwl-runner.metrics DEBUG: EXIT run 1506525376.86 0.883622169495
2017-09-27 15:16:16 arvados.cwl-runner.metrics DEBUG: ENTER jobiter 1506525376.86
2017-09-27 15:16:16 arvados.cwl-runner.metrics DEBUG: EXIT jobiter 1506525376.86 6.29425048828e-05
2017-09-27 15:16:16 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 15:16:16 cwltool INFO: Final process status is success

On qr1hi:

wardv@shell.qr1hi:~$ cwl-runner noreuse.cwl 
2017-09-27 14:50:44 cwltool INFO: /usr/bin/cwl-runner 466e71eb74d1323dc79c8db68d74c5e31e789652 1.0.20170818194607, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170811195303
2017-09-27 14:50:44 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///data-sdc/home/wardv/noreuse.cwl'
2017-09-27 14:50:47 arvados.cwl-runner INFO: Uploaded to qr1hi-4zz18-1mpzig2hh6j19gp
2017-09-27 14:51:01 arvados.cwl-runner INFO: Created pipeline qr1hi-d1hrv-ailajv1w1bv54s5
2017-09-27 14:53:17 arvados.cwl-runner INFO: [job noreuse.cwl] qr1hi-8i9sb-yfp2iy4c5lztlvc is Running
2017-09-27 14:59:03 arvados.cwl-runner INFO: [job noreuse.cwl] qr1hi-8i9sb-yfp2iy4c5lztlvc is Complete
2017-09-27 14:59:04 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 14:59:04 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 14:59:04 cwltool INFO: Final process status is success
wardv@shell.qr1hi:~$ cwl-runner noreuse.cwl 
2017-09-27 14:59:11 cwltool INFO: /usr/bin/cwl-runner 466e71eb74d1323dc79c8db68d74c5e31e789652 1.0.20170818194607, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170811195303
2017-09-27 14:59:11 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///data-sdc/home/wardv/noreuse.cwl'
2017-09-27 14:59:14 arvados.cwl-runner INFO: Using collection qr1hi-4zz18-1mpzig2hh6j19gp
2017-09-27 14:59:15 arvados.cwl-runner INFO: Created pipeline qr1hi-d1hrv-xqiqoz7z58xbc0q
2017-09-27 14:59:15 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 14:59:15 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 14:59:15 cwltool INFO: Final process status is success
wardv@shell.qr1hi:~$ cwl-runner noreuse.cwl 
2017-09-27 15:00:06 cwltool INFO: /usr/bin/cwl-runner 466e71eb74d1323dc79c8db68d74c5e31e789652 1.0.20170818194607, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170811195303
2017-09-27 15:00:06 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///data-sdc/home/wardv/noreuse.cwl'
2017-09-27 15:00:10 arvados.cwl-runner INFO: Using collection qr1hi-4zz18-1mpzig2hh6j19gp
2017-09-27 15:00:10 arvados.cwl-runner INFO: Created pipeline qr1hi-d1hrv-yei99yw56p1atru
2017-09-27 15:00:11 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 15:00:11 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 15:00:11 cwltool INFO: Final process status is success
wardv@shell.qr1hi:~$ vi noreuse.cwl 
wardv@shell.qr1hi:~$ cwl-runner noreuse.cwl 
2017-09-27 15:01:47 cwltool INFO: /usr/bin/cwl-runner 466e71eb74d1323dc79c8db68d74c5e31e789652 1.0.20170818194607, arvados-python-client 0.1.20170818194607, cwltool 1.0.20170811195303
2017-09-27 15:01:47 cwltool INFO: Resolved 'noreuse.cwl' to 'file:///data-sdc/home/wardv/noreuse.cwl'
2017-09-27 15:01:50 arvados.cwl-runner INFO: Using collection qr1hi-4zz18-1mpzig2hh6j19gp
2017-09-27 15:01:51 arvados.cwl-runner INFO: Created pipeline qr1hi-d1hrv-ajm8a2tndcojpj9
2017-09-27 15:01:51 arvados.cwl-runner INFO: Overall process status is success
2017-09-27 15:01:51 arvados.cwl-runner INFO: Final output collection 282fbb8a925551343e98e762fa1ec1be+57
{}
2017-09-27 15:01:51 cwltool INFO: Final process status is success

Subtasks 2 (0 open — 2 closed)

Task #12365: Review 12347-disable-reuseResolvedTom Clegg09/28/2017

Actions
Task #12373: Review 12347-obey-wf-reuse-hintResolvedPeter Amstutz09/28/2017

Actions
Actions
Copy link

Also available in: Atom PDF