Arvados

Scenario:

The API server has some sort of issue where it hangs and eventually times out on one request.

That request is part of a page load for workbench.

Current behavior:

Workbench locks up a passenger thread for a long time, nothing goes into the logs, until finally nginx times out on workbench with a bog standard '504 Gateway Time-out' error to the user (this is bad).

Eventually, workbench's rails code prints out this backtrace followed by the log entry:

#<ArvadosApiClient::NoApiResponseException: HTTPClient::ReceiveTimeoutError error connecting to API server>
/var/www/arvados-workbench/current/app/models/arvados_api_client.rb:149:in `rescue in block in api'
/var/www/arvados-workbench/current/app/models/arvados_api_client.rb:146:in `block in api'
/var/www/arvados-workbench/current/app/models/arvados_api_client.rb:145:in `synchronize'
/var/www/arvados-workbench/current/app/models/arvados_api_client.rb:145:in `api'
/var/www/arvados-workbench/current/app/models/arvados_resource_list.rb:201:in `each_page'
/var/www/arvados-workbench/current/app/models/arvados_resource_list.rb:95:in `results'
/var/www/arvados-workbench/current/app/models/container_work_unit.rb:22:in `children'
/var/www/arvados-workbench/current/app/models/proxy_work_unit.rb:100:in `child_summary'
/var/www/arvados-workbench/current/app/models/proxy_work_unit.rb:150:in `progress'
/var/www/arvados-workbench/current/app/views/projects/_show_dashboard.html.erb:112:in `block in _app_views_projects__show_dashboard_html_erb__3685062099697003217_46369800'
/var/www/arvados-workbench/current/app/views/projects/_show_dashboard.html.erb:77:in `each'
/var/www/arvados-workbench/current/app/views/projects/_show_dashboard.html.erb:77:in `_app_views_projects__show_dashboard_html_erb__3685062099697003217_46369800'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/template.rb:145:in `block in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/notifications.rb:161:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/template.rb:339:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/template.rb:143:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/partial_renderer.rb:306:in `render_partial'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/partial_renderer.rb:279:in `block in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/abstract_renderer.rb:38:in `block in instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/notifications.rb:161:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/abstract_renderer.rb:38:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/partial_renderer.rb:278:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/renderer.rb:47:in `render_partial'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/renderer.rb:21:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/helpers/rendering_helper.rb:32:in `render'
/var/www/arvados-workbench/current/app/views/projects/index.html.erb:6:in `_app_views_projects_index_html_erb___2591413930531012974_46329660'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/template.rb:145:in `block in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/notifications.rb:161:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/template.rb:339:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/template.rb:143:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/template_renderer.rb:55:in `block (2 levels) in render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/abstract_renderer.rb:38:in `block in instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/notifications.rb:161:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/abstract_renderer.rb:38:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/template_renderer.rb:54:in `block in render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/template_renderer.rb:62:in `render_with_layout'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/template_renderer.rb:53:in `render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/template_renderer.rb:17:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/renderer.rb:42:in `render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/renderer/renderer.rb:23:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/rendering.rb:99:in `_render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/streaming.rb:217:in `_render_template'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/rendering.rb:82:in `render_to_body'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/rendering.rb:32:in `render_to_body'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/renderers.rb:32:in `render_to_body'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/abstract_controller/rendering.rb:25:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/rendering.rb:16:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/core_ext/benchmark.rb:12:in `block in ms'
/usr/local/rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/core_ext/benchmark.rb:12:in `ms'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/instrumentation.rb:87:in `cleanup_view_runtime'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activerecord-4.1.12/lib/active_record/railties/controller_runtime.rb:25:in `cleanup_view_runtime'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/instrumentation.rb:43:in `render'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/wiselinks-1.2.1/lib/wiselinks/rendering.rb:34:in `render_with_wiselinks'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:193:in `block (2 levels) in render_index'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/mime_responds.rb:258:in `respond_to'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:176:in `render_index'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:226:in `index'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/abstract_controller/base.rb:189:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:113:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:552:in `block (2 levels) in compile'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:502:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:495:in `block (2 levels) in around'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:286:in `block (2 levels) in halting_and_conditional'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:656:in `require_thread_api_token'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:429:in `block in make_lambda'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:285:in `block in halting_and_conditional'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:494:in `block in around'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:502:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:495:in `block (2 levels) in around'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:306:in `block (2 levels) in halting'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:638:in `set_thread_api_token'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:429:in `block in make_lambda'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:305:in `block in halting'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:494:in `block in around'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:502:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:495:in `block (2 levels) in around'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:306:in `block (2 levels) in halting'
/var/www/arvados-workbench/current/app/controllers/application_controller.rb:568:in `thread_clear'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:429:in `block in make_lambda'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:305:in `block in halting'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:494:in `block in around'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:502:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:86:in `run_callbacks'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/notifications.rb:159:in `block in instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/notifications.rb:159:in `instrument'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activerecord-4.1.12/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/abstract_controller/base.rb:136:in `process'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionview-4.1.12/lib/action_view/rendering.rb:30:in `process'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_controller/metal.rb:232:in `block in action'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/routing/route_set.rb:82:in `dispatch'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/routing/route_set.rb:50:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/journey/router.rb:73:in `block in call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/journey/router.rb:59:in `each'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/journey/router.rb:59:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/routing/route_set.rb:692:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/etag.rb:23:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/conditionalget.rb:25:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/head.rb:11:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/flash.rb:254:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/cookies.rb:562:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activerecord-4.1.12/lib/active_record/query_cache.rb:36:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/callbacks.rb:82:in `run_callbacks'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/lograge-0.3.6/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.1.12/lib/rails/rack/logger.rb:20:in `block in call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/tagged_logging.rb:26:in `tagged'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/tagged_logging.rb:68:in `tagged'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.1.12/lib/rails/rack/logger.rb:20:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/methodoverride.rb:21:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/runtime.rb:17:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/activesupport-4.1.12/lib/active_support/cache/strategy/local_cache_middleware.rb:26:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/sendfile.rb:112:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/actionpack-4.1.12/lib/action_dispatch/middleware/ssl.rb:24:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.1.12/lib/rails/engine.rb:514:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.1.12/lib/rails/application.rb:144:in `call'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.1.12/lib/rails/railtie.rb:194:in `public_send'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.3.0/gems/railties-4.1.12/lib/rails/railtie.rb:194:in `method_missing'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

{"method":"GET","path":"/","format":"html","controller":"projects","action":"index","status":422,"duration":302666.09,"view":30.77,"db":0.0,"params":{"current_request_id":"1493165481-398096792"},"@timestamp":"2017-04-26T00:16:24.500Z","@version":"1","message":"[422] GET / (projects#index)"}

Desired behavior:

a) the user gets a nicer error message, and doesn't have to wait minutes for it to appear. The error comes from workbench, it is not an nginx timeout.

b) the backtrace that is eventually generated, together with the request that caused the problem, is set aside or shipped to a preconfigured address for the attention of the ops team. The ops team should be notified somehow.