Arvados

It appears that when certain API server requests fail, the internal state of FUSE is left in an inconsistent state that results in a crash.

2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr WARNING:root:Retry #1 for request: GET https://su92l.arvadosapi.com/arvados/v1/collections/69414010a3d0f286ad6eb5a578801aa1%2B11278592?alt=json, following status: 503
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr WARNING:root:Retry #2 for request: GET https://su92l.arvadosapi.com/arvados/v1/collections/69414010a3d0f286ad6eb5a578801aa1%2B11278592?alt=json, following status: 503
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr WARNING:root:Retry #3 for request: GET https://su92l.arvadosapi.com/arvados/v1/collections/69414010a3d0f286ad6eb5a578801aa1%2B11278592?alt=json, following status: 503
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr 2015-07-08 23:35:20 arvados.arvados_fuse[31747] ERROR: arv-mount 69414010a3d0f286ad6eb5a578801aa1+11278592: error
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr Traceback (most recent call last):
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados_fuse/fusedir.py", line 404, in update
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     num_retries=self.num_retries)
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1616, in __init__
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     super(CollectionReader, self).__init__(manifest_locator_or_text, *args, **kwargs)
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1188, in __init__
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     self._populate()
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1306, in _populate
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     error_via_keep))
2015-07-08_23:37:55 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr NotFoundError: Failed to retrieve collection '69414010a3d0f286ad6eb5a578801aa1+11278592' from either API server (<HttpError 503 when requesting https://su92l.arvadosapi.com/arvados/v1/collec
tions/69414010a3d0f286ad6eb5a578801aa1%2B11278592?alt=json returned "Service Unavailable">) or Keep (failed to read 69414010a3d0f286ad6eb5a578801aa1+11278592: no Keep services available (<HttpError 503 when requesting https://su92l.arvadosapi.com/arvados
/v1/keep_services/accessible?alt=json returned "Service Unavailable">)).

2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr ERROR:arvados.arvados_fuse:Unhandled exception during FUSE operation
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr Traceback (most recent call last):
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados_fuse/__init__.py", line 253, in catch_exceptions_wrapper
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     return orig_func(self, *args, **kwargs)
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados_fuse/__init__.py", line 526, in opendir
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     self.inodes.touch(p)
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados_fuse/__init__.py", line 218, in touch
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     self.inode_cache.touch(entry)
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados_fuse/__init__.py", line 180, in touch
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     self._remove(obj, False)
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados_fuse/__init__.py", line 150, in _remove
2015-07-08_23:37:56 su92l-8i9sb-2sy4hr0v4pgr4rn 24832 15 stderr     del self._by_uuid[obj.cache_uuid]