Bug #11917
closed
Error fetching collection (HttpError 422) during call to arvados.util.listdir_recursive
Added by Joshua Randall over 7 years ago.
Updated over 6 years ago.
Estimated time:
(Total: 0.00 h)
Description
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr 2017-06-27 16:17:34 arvados.arvados_fuse[45815] ERROR: Error fetching collection 'f23f6b30aa7d658b565af6d4e2
e5368a+9409': Failed to retrieve collection 'f23f6b30aa7d658b565af6d4e2e5368a+9409' from either API server (<HttpError 422 when requesting https://api.arvados.sanger.ac.uk/arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409?alt=json returned "#<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/www/arvados-api/current/tmp/cache/94F/070/.permissions_check.70302752543300.5829.180478>">) or Keep (f23f6b30aa7d658b565af6d4e2e5368a+9409 not found: http://humgen-01-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-03-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-04-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-02-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-03-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-02-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-03-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-01-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-02-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-04-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-04-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ; http://humgen-01-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ).
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr Traceback (most recent call last):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr File "/data/crunch-tmp/crunch-job/src/crunch_scripts/gatk-haplotypecaller-cram.py", line 99, in <module>
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr main()
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr File "/data/crunch-tmp/crunch-job/src/crunch_scripts/gatk-haplotypecaller-cram.py", line 54, in main
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr cram_file = gatk_helper.mount_gatk_cram_input(input_param="input")
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr File "/data/crunch-tmp/crunch-job/src/crunch_scripts/hgi_arvados/gatk_helper.py", line 81, in mount_gatk_cram_input
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr for f in arvados.util.listdir_recursive(input_dir):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr File "/usr/lib/python2.7/dist-packages/arvados/util.py", line 332, in listdir_recursive
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr for ent in sorted(os.listdir(dirname)):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr OSError: [Errno 2] No such file or directory: '/keep/f23f6b30aa7d658b565af6d4e2e5368a+9409'
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 child 29639 on humgen-02-01.16 exit 1 success=
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 failure (#1, permanent) after 35 seconds
From production.log:
[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/www/arvados-api/current/tmp/cache/94F/070/.permissions_check.70302752543300.5829.180478>
[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] /usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `initialize'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `open'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `rescue in block in touch'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1162:in `block in touch'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1160:in `each'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1160:in `touch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/core_ext/file/atomic.rb:58:in `probe_stat_in'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/core_ext/file/atomic.rb:32:in `atomic_write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/file_store.rb:83:in `write_entry'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/strategy/local_cache.rb:115:in `write_entry'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:391:in `block in write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:547:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:166:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:547:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:389:in `write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:588:in `save_block_result_to_cache'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:299:in `fetch'
/var/www/arvados-api/current/lib/sweep_trashed_collections.rb:21:in `sweep_if_stale'
/var/www/arvados-api/current/app/models/collection.rb:441:in `where'
/var/www/arvados-api/current/app/controllers/arvados/v1/collections_controller.rb:28:in `find_object_by_uuid'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:432:in `block in make_lambda'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:145:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:145:in `block in halting_and_conditional'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:137:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionview-4.2.5.2/lib/action_view/rendering.rb:30:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:237:in `block in action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:815:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/builder.rb:63:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/etag.rb:24:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/conditionalget.rb:25:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/query_cache.rb:36:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/lograge-0.4.1/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:20:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:26:in `tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:68:in `tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:20:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/methodoverride.rb:22:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/runtime.rb:18:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/sendfile.rb:113:in `call'
/var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:59:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/engine.rb:518:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/application.rb:165:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `public_send'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `method_missing'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] Error 1498576650+aad9240b: 422
[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] {"method":"GET","path":"/arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409","format":"json","control
ler":"Arvados::V1::CollectionsController","action":"show","status":422,"duration":4.1,"view":0.61,"db":0.0,"params":{"alt":"json"},"@timestamp":"2017-06-27T15:17:30.34520886
4Z","@version":"1","message":"[422] GET /arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409 (Arvados::V1::CollectionsController#show)"}
This code looks like a supported use of Rails cache:
Rails.cache.fetch('SweepTrashedCollections', expires_in: exp) do
need = true
end
In activesupport-4.2.5.2/lib/active_support/cache/file_store.rb
def write_entry(key, entry, options)
file_name = key_file_path(key)
return false if options[:unless_exist] && File.exist?(file_name)
ensure_cache_path(File.dirname(file_name))
File.atomic_write(file_name, cache_path) {|f| Marshal.dump(entry, f)}
true
end
Perhaps empty cache directories are being deleted between "ensure_cache_path" and "File.atomic_write"?
We do this in source:services/api/config/environment.rb:
# Initialize the rails application
Server::Application.initialize!
begin
Rails.cache.clear
rescue Errno::ENOENT => e
# Cache directory does not exist? Then cache is clear, proceed.
Rails.logger.warn "In Rails.cache.clear, ignoring #{e.inspect}"
end
I'm guessing this is what's clearing the cache directories. It's possible this is running every time Passenger brings up a new worker. It's very likely it is also running every time a rake task is invoked by cron. We should fix this so it only runs once at server startup, and never on periodic rake tasks... or perhaps so it doesn't run at all.
- Category set to API
- Status changed from New to In Progress
- Assigned To set to Tom Clegg
- Target version set to 2017-07-19 sprint
We use expiry times and a finite number of keys, so the cache shouldn't grow indefinitely.
We should add a cache-clearing thing that does "delete all entries matching *" instead of the non-race-safe "clear" method. And choose a better moment to run it, so it doesn't clear every time someone runs a rails-based maintenance cron job...
"expiry times" == "clear out old entries"
Perhaps we could put a rake tmp:cache:clear in the API server boot script if we're really concerned about stale cache space from e.g. a server crash, but otherwise it sounds like the combination of limited keys and expiry times should do the trick.
- Status changed from In Progress to Feedback
- Target version changed from 2017-07-19 sprint to Arvados Future Sprints
- Story points set to 0.0
- Status changed from Feedback to Resolved
- Story points deleted (
0.0)
- Target version deleted (
Arvados Future Sprints)
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by Tom Morris 
Updated by Radhika Chippada