Bug #11917: Error fetching collection (HttpError 422) during call to arvados.util.listdir_recursive - Arvados

Actions

Copy link

Bug #11917

closed

Error fetching collection (HttpError 422) during call to arvados.util.listdir_recursive

Added by Joshua Randall over 7 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Tom Clegg

Category:

API

Target version:

Start date:

06/29/2017

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Story points:

Description

2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr 2017-06-27 16:17:34 arvados.arvados_fuse[45815] ERROR: Error fetching collection 'f23f6b30aa7d658b565af6d4e2
e5368a+9409': Failed to retrieve collection 'f23f6b30aa7d658b565af6d4e2e5368a+9409' from either API server (<HttpError 422 when requesting https://api.arvados.sanger.ac.uk/arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409?alt=json returned "#<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/www/arvados-api/current/tmp/cache/94F/070/.permissions_check.70302752543300.5829.180478>">) or Keep (f23f6b30aa7d658b565af6d4e2e5368a+9409 not found:  http://humgen-01-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-03-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-04-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-02-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-03-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-02-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-03-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-01-03.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-02-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-04-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-04-01.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ;  http://humgen-01-02.internal.sanger.ac.uk:25107/ responded with 404 HTTP/1.1 404 Not Found\015
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr ).
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr Traceback (most recent call last):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr   File "/data/crunch-tmp/crunch-job/src/crunch_scripts/gatk-haplotypecaller-cram.py", line 99, in <module>
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr     main()
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr   File "/data/crunch-tmp/crunch-job/src/crunch_scripts/gatk-haplotypecaller-cram.py", line 54, in main
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr     cram_file = gatk_helper.mount_gatk_cram_input(input_param="input")
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr   File "/data/crunch-tmp/crunch-job/src/crunch_scripts/hgi_arvados/gatk_helper.py", line 81, in mount_gatk_cram_input
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr     for f in arvados.util.listdir_recursive(input_dir):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr   File "/usr/lib/python2.7/dist-packages/arvados/util.py", line 332, in listdir_recursive
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr     for ent in sorted(os.listdir(dirname)):
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 stderr OSError: [Errno 2] No such file or directory: '/keep/f23f6b30aa7d658b565af6d4e2e5368a+9409'
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 child 29639 on humgen-02-01.16 exit 1 success=
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2017-06-27_15:17:34 z8ta6-8i9sb-b6xy0airw233wkt 21286 151 failure (#1, permanent) after 35 seconds

Subtasks 1 (0 open — 1 closed)

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Joshua Randall over 7 years ago

From production.log:

[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/www/arvados-api/current/tmp/cache/94F/070/.permissions_check.70302752543300.5829.180478>
[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] /usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `initialize'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `open'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1166:in `rescue in block in touch'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1162:in `block in touch'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1160:in `each'
/usr/local/lib/ruby/2.1.0/fileutils.rb:1160:in `touch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/core_ext/file/atomic.rb:58:in `probe_stat_in'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/core_ext/file/atomic.rb:32:in `atomic_write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/file_store.rb:83:in `write_entry'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/strategy/local_cache.rb:115:in `write_entry'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:391:in `block in write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:547:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:166:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:547:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:389:in `write'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:588:in `save_block_result_to_cache'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:299:in `fetch'
/var/www/arvados-api/current/lib/sweep_trashed_collections.rb:21:in `sweep_if_stale'
/var/www/arvados-api/current/app/models/collection.rb:441:in `where'
/var/www/arvados-api/current/app/controllers/arvados/v1/collections_controller.rb:28:in `find_object_by_uuid'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:432:in `block in make_lambda'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:145:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:145:in `block in halting_and_conditional'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:504:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `instrument'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:137:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionview-4.2.5.2/lib/action_view/rendering.rb:30:in `process'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:237:in `block in action'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:815:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/omniauth-1.4.2/lib/omniauth/builder.rb:63:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/etag.rb:24:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/conditionalget.rb:25:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/query_cache.rb:36:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/lograge-0.4.1/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:20:in `block in call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:26:in `tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/tagged_logging.rb:68:in `tagged'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:20:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/methodoverride.rb:22:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/runtime.rb:18:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/rack-1.6.5/lib/rack/sendfile.rb:113:in `call'
/var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:59:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/engine.rb:518:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/application.rb:165:in `call'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `public_send'
/var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `method_missing'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] Error 1498576650+aad9240b: 422
[api.arvados.sanger] [8a7ec0e5-73f0-445d-aa8b-e7bf4442c444] {"method":"GET","path":"/arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409","format":"json","control
ler":"Arvados::V1::CollectionsController","action":"show","status":422,"duration":4.1,"view":0.61,"db":0.0,"params":{"alt":"json"},"@timestamp":"2017-06-27T15:17:30.34520886
4Z","@version":"1","message":"[422] GET /arvados/v1/collections/f23f6b30aa7d658b565af6d4e2e5368a%2B9409 (Arvados::V1::CollectionsController#show)"}

Actions

Copy link

Updated by Tom Clegg over 7 years ago

This code looks like a supported use of Rails cache:

    Rails.cache.fetch('SweepTrashedCollections', expires_in: exp) do
      need = true
    end

Actions

Copy link

Updated by Tom Clegg over 7 years ago

In activesupport-4.2.5.2/lib/active_support/cache/file_store.rb

        def write_entry(key, entry, options)
          file_name = key_file_path(key)
          return false if options[:unless_exist] && File.exist?(file_name)
          ensure_cache_path(File.dirname(file_name))
          File.atomic_write(file_name, cache_path) {|f| Marshal.dump(entry, f)}
          true
        end

Perhaps empty cache directories are being deleted between "ensure_cache_path" and "File.atomic_write"?

Actions

Copy link

Updated by Tom Clegg over 7 years ago

We do this in source:services/api/config/environment.rb:

# Initialize the rails application
Server::Application.initialize!
begin
  Rails.cache.clear
rescue Errno::ENOENT => e
  # Cache directory does not exist? Then cache is clear, proceed.
  Rails.logger.warn "In Rails.cache.clear, ignoring #{e.inspect}" 
end

I'm guessing this is what's clearing the cache directories. It's possible this is running every time Passenger brings up a new worker. It's very likely it is also running every time a rake task is invoked by cron. We should fix this so it only runs once at server startup, and never on periodic rake tasks... or perhaps so it doesn't run at all.

Actions

Copy link

Updated by Tom Clegg over 7 years ago

11917-dont-clear-cache @ e8d11e963248fada51bdb1ecc59446adccff440f

Actions

Copy link

Updated by Tom Clegg over 7 years ago

Category set to API
Status changed from New to In Progress
Assigned To set to Tom Clegg

Actions

Copy link

Updated by Tom Morris over 7 years ago

Target version set to 2017-07-19 sprint

Actions

Copy link

Updated by Radhika Chippada over 7 years ago

Is the cache LRU? http://guides.rubyonrails.org/caching_with_rails.html says: "As the cache will grow until the disk is full, it is recommended to periodically clear out old entries." But I couldn't tell how / if this is being done in our server.

Actions

Copy link

Updated by Tom Clegg over 7 years ago

We use expiry times and a finite number of keys, so the cache shouldn't grow indefinitely.

We should add a cache-clearing thing that does "delete all entries matching *" instead of the non-race-safe "clear" method. And choose a better moment to run it, so it doesn't clear every time someone runs a rails-based maintenance cron job...

Actions

Copy link

#10

Updated by Tom Morris over 7 years ago

"expiry times" == "clear out old entries"

Perhaps we could put a rake tmp:cache:clear in the API server boot script if we're really concerned about stale cache space from e.g. a server crash, but otherwise it sounds like the combination of limited keys and expiry times should do the trick.

Actions

Copy link

#11