Arvados

Oldest 'OK' run: https://ci.curoverse.com/job/qr1hi-automated-performance-suite/3162/

Since then (ie Dec. 26, 2016) they have been broken. For example, from https://ci.curoverse.com/job/qr1hi-automated-performance-suite/3427/console:

                       ********** Starting performance test **********

Previous HEAD position was aaa45b0... Merge branch '10968-arvput-resume-progress' Closes #10968
HEAD is now at 85f8ab2... Bump up rails package iteration.
Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
Using rake 10.4.2
Using RedCloth 4.2.9
Using i18n 0.7.0
Using json 1.8.3
Using minitest 5.8.4
Using thread_safe 0.3.5
Using builder 3.2.2
Using erubis 2.7.0
Using rack 1.5.5
Using mime-types 2.99
Using arel 5.0.1.20140414130214
Using addressable 2.4.0
Using andand 1.3.3
Using angularjs-rails 1.3.8
Using extlib 0.9.16
Using multi_json 1.12.1
Using multipart-post 2.0.0
Using jwt 1.5.4
Using little-plugger 1.1.4
Using memoist 0.14.0
Using os 0.9.6
Using retriable 1.4.1
Using sass 3.4.9
Using thor 0.19.1
Using columnize 0.9.0
Using debugger-linecache 1.2.0
Using slop 3.6.0
Using highline 1.6.21
Using net-ssh 2.9.2
Using mini_portile 0.6.2
Using ffi 1.9.10
Using cliver 0.3.2
Using coffee-script-source 1.8.0
Using execjs 2.7.0
Using commonjs 0.2.7
Using daemon_controller 1.2.0
Using deep_merge 1.0.1
Using docile 1.1.5
Using headless 1.0.2
Using httpclient 2.8.2.4
Using libv8 3.16.14.7
Using logstash-event 1.2.02
Using metaclass 0.0.4
Using oj 2.11.2
Using bundler 1.11.2
Using websocket-extensions 0.1.1
Using rails-perftest 0.0.5
Using raphael-rails 2.1.2
Using ref 1.0.5
Using ruby-debug-passenger 0.2.0
Using ruby-prof 0.15.2
Using rubyzip 1.1.7
Using safe_yaml 1.0.4
Using tilt 1.4.1
Using websocket 1.2.2
Using simplecov-html 0.8.0
Using sshkey 1.6.1
Using wiselinks 1.2.1
Using rake-compiler 0.9.5
Using tzinfo 1.2.2
Using rack-test 0.6.3
Using sprockets 3.2.0
Using rack-mini-profiler 0.9.2
Using mail 2.6.3
Using launchy 2.4.3
Using autoparse 0.3.3
Using faraday 0.9.2
Using logging 2.1.0
Using bootstrap-sass 3.1.1.1
Using byebug 3.5.1
Using net-scp 1.2.1
Using net-sftp 2.1.2
Using net-ssh-gateway 1.2.0
Using nokogiri 1.6.6.4
Using childprocess 0.5.6
Using coffee-script 2.3.0
Using uglifier 2.7.2
Using less 2.6.0
Using passenger 4.0.57
Using mocha 1.1.0
Using websocket-driver 0.5.1
Using therubyracer 0.12.1
Using simplecov 0.9.1
Using fast_stack 0.1.0
Using activesupport 4.1.12
Using signet 0.7.2
Using capistrano 2.15.5
Using xpath 2.0.0
Using selenium-webdriver 2.53.1
Using simplecov-rcov 0.2.3
Using flamegraph 0.1.0
Using actionview 4.1.12
Using activemodel 4.1.12
Using googleauth 0.5.1
Using rvm-capistrano 1.5.5
Using capybara 2.5.0
Using actionpack 4.1.12
Using activerecord 4.1.12
Using google-api-client 0.8.6
Using poltergeist 1.5.1
Using actionmailer 4.1.12
Using railties 4.1.12
Using less-rails 2.6.0
Using sprockets-rails 2.3.2
Using activerecord-nulldb-adapter 0.3.1
Using arvados 0.1.20160420143004
Using bootstrap-tab-history-rails 0.1.0
Using bootstrap-x-editable-rails 1.5.1.1
Using coffee-rails 4.1.0
Using jquery-rails 3.1.2
Using lograge 0.3.6
Using morrisjs-rails 0.5.1
Using rails 4.1.12
Using sass-rails 5.0.1
Using piwik_analytics 1.0.2
Using themes_for_rails 0.5.1 from https://github.com/curoverse/themes_for_rails (at master@6115487)
Bundle complete! 46 Gemfile dependencies, 116 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Called 'load' without the :safe option -- defaulting to safe mode.
You can avoid this warning in the future by setting the SafeYAML::OPTIONS[:default_mode] option (to :safe or :unsafe).
Called 'load' without the :safe option -- defaulting to safe mode.
You can avoid this warning in the future by setting the SafeYAML::OPTIONS[:default_mode] option (to :safe or :unsafe).
Using display :182 for headless tests
Using port 32800 for poltergeist
Run options: --seed 41450

# Running:

BrowsingTest#test_home_page (5.93 sec warmup)
           wall_time: 5.06 sec
.Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g
Saved ./tmp/workbench-fail-1.png
BrowsingTest#test_search_for_hash (0 ms warmup)
Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g
Saved ./tmp/workbench-fail-2.png
Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g
Saved ./tmp/workbench-fail-3.png
Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g
Saved ./tmp/workbench-fail-4.png
Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g
Saved ./tmp/workbench-fail-5.png
Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g
Saved ./tmp/workbench-fail-6.png
           wall_time: 0 ms
E

Finished in 445.442299s, 0.0045 runs/s, 0.0000 assertions/s.

  1) Error:
BrowsingTest#test_search_for_hash:
Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches
    test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>'
    test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>'
    test/test_helper.rb:299:in `run'

Error:
BrowsingTest#test_search_for_hash:
Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches
    test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>'
    test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>'
    test/test_helper.rb:299:in `run'

Error:
BrowsingTest#test_search_for_hash:
Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches
    test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>'
    test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>'
    test/test_helper.rb:299:in `run'

Error:
BrowsingTest#test_search_for_hash:
Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches
    test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>'
    test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>'
    test/test_helper.rb:299:in `run'

Error:
BrowsingTest#test_search_for_hash:
Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches
    test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>'
    test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>'
    test/test_helper.rb:299:in `run'

Error:
BrowsingTest#test_search_for_hash:
Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches
    test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>'
    test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>'
    test/test_helper.rb:299:in `run'

2 runs, 0 assertions, 0 failures, 1 errors, 0 skips
Coverage report generated for Unit Tests to /data/1/jenkins/workspace/qr1hi-automated-performance-suite/apps/workbench/coverage. 198 / 473 LOC (41.86%) covered.
Coverage report Rcov style generated for Unit Tests to /data/1/jenkins/workspace/qr1hi-automated-performance-suite/apps/workbench/coverage/rcov
rake aborted!
Command failed with status (1): [ruby -I"lib:test" -I"/usr/local/rvm/gems/ruby-2.1.5/gems/rake-10.4.2/lib" "/usr/local/rvm/gems/ruby-2.1.5/gems/rake-10.4.2/lib/rake/rake_test_loader.rb" "test/performance/**/*_test.rb" ]
/usr/local/rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => test:benchmark
(See full trace by running task with --trace)
Previous HEAD position was 85f8ab2... Bump up rails package iteration.
Switched to branch 'master'
Your branch is behind 'origin/master' by 4870 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

The error about websockets is a bit mysterious but also probably a red herring. It may be related to the older version of phantomjs (1.9.8) we are pinned to, at least according to this page: http://www.howtobuildsoftware.com/index.php/how-do/7o0/amazon-web-services-meteor-websocket-phantomjs-error-during-websocket-handshake-location-mismatch-wss-ws-

I verified in the logs on qr1hi that the wss:// service is the one being contacted during the test (so it's not trying to connect to the non-tls version).

The first diagnostics test that fails (BrowsingTest#test_search_for_hash) could be a legitimate failure: doing the same thing (searching for the word 'hash' in the search box) takes 1.3 minutes according to my browser's debug console.

Suggested actions:

a) upgrade workbench's bundle to see if that helps with the wss vs ws issue
b) fix search performance on qr1hi (for reference, the same search takes just 33s on su92l)