Arvados

https://cloud.curoverse.com/pipeline_instances/qr1hi-d1hrv-72yc2r651sxbia0#Log

2015-10-16_19:48:35 Traceback (most recent call last):
2015-10-16_19:48:35 File "/usr/local/bin/arv-get", line 207, in <module>
2015-10-16_19:48:35 for data in f.readall():
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/arvfile.py", line 100, in readall
2015-10-16_19:48:35 data = self.read(size, num_retries=num_retries)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/arvfile.py", line 51, in before_close_wrapper
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/arvfile.py", line 207, in read
2015-10-16_19:48:35 num_retries=num_retries)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 85, in readfrom
2015-10-16_19:48:35 data.append(self._keepget(lr.locator, num_retries=num_retries)[lr.segment_offset:lr.segment_offset+lr.segment_size])
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 74, in _keepget
2015-10-16_19:48:35 return self._keep.get(locator, num_retries=num_retries)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 907, in get
2015-10-16_19:48:35 Traceback (most recent call last):
2015-10-16_19:48:35 File "/usr/local/bin/arv-get", line 207, in <module>
2015-10-16_19:48:35 for data in f.readall():
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/arvfile.py", line 100, in readall
2015-10-16_19:48:35 data = self.read(size, num_retries=num_retries)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/arvfile.py", line 51, in before_close_wrapper
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/arvfile.py", line 207, in read
2015-10-16_19:48:35 num_retries=num_retries)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 85, in readfrom
2015-10-16_19:48:35 data.append(self._keepget(lr.locator, num_retries=num_retries)[lr.segment_offset:lr.segment_offset+lr.segment_size])
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 74, in _keepget
2015-10-16_19:48:35 return self._keep.get(locator, num_retries=num_retries)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
2015-10-16_19:48:35 return orig_func(self, *args, **kwargs)
2015-10-16_19:48:35 File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 907, in get
2015-10-16_19:48:35 "failed to read {}".format(loc_s), service_errors, label="service")
2015-10-16_19:48:35 arvados.errors.KeepReadError: failed to read 76bff9ed2fa9214129744361101a633b+67108864+A411c1b48ee3e34adeb6bc011e9a98454bf9399fb@5633c805: service http://keep13.qr1hi.arvadosapi.com:25107/ responded with 0 (7, 'Failed to connect to keep13.qr1hi.arvadosapi.com port 25107: Connection refused'); service http://keep11.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep16.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep15.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep12.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep10.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep14.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35
2015-10-16_19:48:35 "failed to read {}".format(loc_s), service_errors, label="service")
2015-10-16_19:48:35 arvados.errors.KeepReadError: failed to read 76bff9ed2fa9214129744361101a633b+67108864+Abde11cf8810b3406ad00cdc1e300377de03c9bff@5633c800: service http://keep13.qr1hi.arvadosapi.com:25107/ responded with 0 (7, 'Failed to connect to keep13.qr1hi.arvadosapi.com port 25107: Connection refused'); service http://keep11.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep16.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep15.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep12.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep10.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35 ; service http://keep14.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
2015-10-16_19:48:35
2015-10-16_19:48:35 Error response from daemon: Untar re-exec error: exit status 1: output: unexpected EOF
2015-10-16_19:48:35 srun: error: compute2: task 1: Exited with exit code 1
2015-10-16_19:48:35 Error response from daemon: Untar re-exec error: exit status 1: output: unexpected EOF
2015-10-16_19:48:35 srun: error: compute0: task 0: Exited with exit code 1
2015-10-16_19:48:36 qr1hi-8i9sb-s4v2o77d28kius0 46932 Installing Docker image from e72de48b6dd529bd8960246cea4b16fb+966 exited 1 at /usr/local/arvados/src/sdk/cli/bin/crunch-job line 442
2015-10-16_19:48:36 qr1hi-8i9sb-s4v2o77d28kius0 46932 Freeze not implemented
2015-10-16_19:48:36 qr1hi-8i9sb-s4v2o77d28kius0 46932 collate
2015-10-16_19:48:36 qr1hi-8i9sb-s4v2o77d28kius0 46932 collated output manifest text to send to API server is 0 bytes with access tokens
2015-10-16_19:48:38 qr1hi-8i9sb-s4v2o77d28kius0 46932 log collection is dd3273de25ddb275246bdc328ba8536f+81
2015-10-16_19:48:40 Died at /usr/local/arvados/src/sdk/cli/bin/crunch-job line 1707, <DATA> line 1.