Arvados

• ecfc7ea60 15044: Specify test database collation.
• 5005c8c8d 15044: Fix unnecessarily unsafe yaml load.
• 1bd29992b 15044: Fix workbench test args in interactive mode.
• 6c24b6ea6 15044: Improve error reporting in test case.
• 8ed7de424 15044: Listen on localhost instead of default dual-stack [::].
• d12e4ed76 15044: Test gofmt as a separate suite, instead of refusing to run Go tests.
• 7290838a2 15044: Always use the en_US.UTF-8 locale. Add sanity check.
• 2614d2fc2 15044: Fix error messages.
• c1794d8ca 15044: Fix interactive command handling.
• 4398ef036 15044: Override unusable nginx defaults. Test nginx startup success.

• 752674a54 15044: Fix sensitivity to group existence on the testing host.
  (services/login-sync)
• 53647be00 15044: Don't kill run-tests's keepproxy server in sdk/python tests.
  (before this, "run all tests" would fail the apps/workbench_integration test that uses keepproxy, because keepproxy had been killed during the sdk/python tests)

I ran the entire test suite in arvbox:

Fail: services/api tests (304s)
Fail: gofmt tests (1s)
Fail: doc tests (77s)
Fail: services/login-sync tests (1s)
Fail: services/nodemanager tests (12s)

services/api¶

  1) Failure:
Arvados::V1::SchemaControllerTest#test_discovery_document_fields [/usr/src/arvados/services/api/test/functional/arvados/v1/schema_controller_test.rb:35]:
Expected /^unknown$/ to match "1.3.1.20190326215235".

1887 runs, 11498 assertions, 1 failures, 0 errors, 0 skips
SimpleCov failed to recognize the test framework and/or suite used. Please specify manually using SimpleCov.command_name 'Unit Tests'.
Coverage report generated for Unit Tests, Unknown Test Framework to /usr/src/arvados/services/api/coverage. 5505 / 6334 LOC (86.91%) covered.
Coverage report Rcov style generated for Unit Tests, Unknown Test Framework to /usr/src/arvados/services/api/coverage/rcov
======= services/api tests -- FAILED

gofmt¶

======= test gofmt
diff -u lib/cloud/ec2/ec2.go.orig lib/cloud/ec2/ec2.go
--- lib/cloud/ec2/ec2.go.orig    2019-04-01 17:54:29.466044177 +0000
+++ lib/cloud/ec2/ec2.go    2019-04-01 17:54:29.470044220 +0000
@@ -191,7 +191,7 @@
             }},
         DisableApiTermination:             aws.Bool(false),
         InstanceInitiatedShutdownBehavior: aws.String("terminate"),
-        UserData: aws.String(base64.StdEncoding.EncodeToString([]byte("#!/bin/sh\n" + initCommand + "\n"))),
+        UserData:                          aws.String(base64.StdEncoding.EncodeToString([]byte("#!/bin/sh\n" + initCommand + "\n"))),
         TagSpecifications: []*ec2.TagSpecification{
             &ec2.TagSpecification{
                 ResourceType: aws.String("instance"),
diff -u sdk/go/keepclient/keepclient_test.go.orig sdk/go/keepclient/keepclient_test.go
--- sdk/go/keepclient/keepclient_test.go.orig    2019-04-01 17:54:29.698046652 +0000
+++ sdk/go/keepclient/keepclient_test.go    2019-04-01 17:54:29.698046652 +0000
@@ -695,13 +695,13 @@
             "zzzzz-bi6l4-yyyyyyyyyyyyyyy": ks0.url,
             "zzzzz-bi6l4-xxxxxxxxxxxxxxx": ks0.url,
             "zzzzz-bi6l4-wwwwwwwwwwwwwww": ks0.url,
-            uuid: ks.url},
+            uuid:                          ks.url},
         nil,
         map[string]string{
             "zzzzz-bi6l4-yyyyyyyyyyyyyyy": ks0.url,
             "zzzzz-bi6l4-xxxxxxxxxxxxxxx": ks0.url,
             "zzzzz-bi6l4-wwwwwwwwwwwwwww": ks0.url,
-            uuid: ks.url},
+            uuid:                          ks.url},
     )

     r, n, uri, err := kc.Get(hash + "+K@" + uuid)
======= gofmt tests -- FAILED

doc¶

URL        `R.css'
Parent URL file:///usr/src/arvados/doc/.site/sdk/R/arvados/workflows.update.html, line 3, col 1
Real URL   file:///usr/src/arvados/doc/.site/sdk/R/arvados/R.css
Check time 0.001 seconds
Result     Error: URLError: <urlopen error [Errno 2] No such file or directory: '/usr/src/arvados/doc/.site/sdk/R/arvados/R.css'>
 7 threads active,  2784 links queued, 18546 links in 679 URLs checked, runtime 26 seconds

URL        `resources/fonts/dejavu.css'
Parent URL file:///usr/src/arvados/doc/.site/sdk/java-v2/javadoc/stylesheet.css, line 4, col 12
Real URL   file:///usr/src/arvados/doc/.site/sdk/java-v2/javadoc/resources/fonts/dejavu.css
Check time 0.001 seconds
Result     Error: URLError: <urlopen error [Errno 2] No such file or directory: '/usr/src/arvados/doc/.site/sdk/java-v2/javadoc/resources/fonts/dejavu.css'>
10 threads active,  1199 links queued, 24050 links in 739 URLs checked, runtime 31 seconds

Statistics:
Downloaded: 10.5MB.
Content types: 524 image, 25006 text, 0 video, 0 audio, 854 application, 2 mail and 1139 other.
URL lengths: min=17, max=143, avg=74.

That's it. 27525 links in 789 URLs checked. 0 warnings found. 173 errors found.
Stopped checking at 2019-04-01 17:55:46+000 (34 seconds)
======= doc tests -- FAILED

services/login-sync¶

======= test services/login-sync
/usr/src/arvados/services/login-sync/test/stubs.rb:6:in `require': cannot load such file -- mocha/minitest (LoadError)
    from /usr/src/arvados/services/login-sync/test/stubs.rb:6:in `<top (required)>'
    from /usr/src/arvados/services/login-sync/test/test_add_user.rb:7:in `require'
    from /usr/src/arvados/services/login-sync/test/test_add_user.rb:7:in `<top (required)>'
    from /var/lib/gems/gems/rake-12.0.0/lib/rake/rake_test_loader.rb:15:in `require'
    from /var/lib/gems/gems/rake-12.0.0/lib/rake/rake_test_loader.rb:15:in `block in <main>'
    from /var/lib/gems/gems/rake-12.0.0/lib/rake/rake_test_loader.rb:4:in `select'
    from /var/lib/gems/gems/rake-12.0.0/lib/rake/rake_test_loader.rb:4:in `<main>'
rake aborted!
Command failed with status (1)
/var/lib/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/cli/exec.rb:75:in `load'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/cli/exec.rb:75:in `kernel_load'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/cli/exec.rb:28:in `run'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/cli.rb:424:in `exec'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/cli.rb:27:in `dispatch'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/cli.rb:18:in `start'
/var/lib/gems/gems/bundler-1.16.1/exe/bundle:30:in `block in <top (required)>'
/var/lib/gems/gems/bundler-1.16.1/lib/bundler/friendly_errors.rb:122:in `with_friendly_errors'
/var/lib/gems/gems/bundler-1.16.1/exe/bundle:22:in `<top (required)>'
/var/lib/gems/bin/bundle:22:in `load'
/var/lib/gems/bin/bundle:22:in `<main>'
Tasks: TOP => test
(See full trace by running task with --trace)
======= services/login-sync tests -- FAILED
======= test services/login-sync -- 1s

services/nodemanager¶

======================================================================
FAIL: test_new_node_when_booted_node_not_usable (tests.test_daemon.NodeManagerDaemonActorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/arvados/services/nodemanager/tests/test_daemon.py", line 374, in test_new_node_when_booted_node_not_usable
    self.assertEqual(2, self.node_setup.start.call_count)
AssertionError: 2 != 1

----------------------------------------------------------------------
Ran 274 tests in 10.703s

FAILED (failures=1)
Test failed: <unittest.runner.TextTestResult run=274 errors=0 failures=1>
error: Test failed: <unittest.runner.TextTestResult run=274 errors=0 failures=1>
======= services/nodemanager tests -- FAILED

$ arvbox start test --skip-install --only services/nodemanager
R SDK not needed, it will not be installed.
Checking dependencies:
locale: en_US.UTF-8
virtualenv: 15.1.0
ruby: ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-linux]
go: go version go1.11.5 linux/amd64
gcc: gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
fuse.h: /usr/include/fuse/fuse.h
gnutls.h: /usr/include/gnutls/gnutls.h
Python2 pyconfig.h: /usr/include/python2.7/pyconfig.h
Python3 pyconfig.h: /usr/include/python3.5m/pyconfig.h
nginx: nginx version: nginx/1.10.3
perl: This is perl 5, version 24, subversion 1 (v5.24.1) built for x86_64-linux-gnu-thread-multi
perl ExtUtils::MakeMaker: 7.1002
perl JSON: 2.90
perl LWP: 6.15
perl Net::SSL: 2.88
gitolite: /usr/bin/gitolite
npm: 6.4.1
cadaver: cadaver 0.23.3
libattr1 xattr.h: /usr/include/attr/xattr.h
libcurl curl.h: /usr/include/x86_64-linux-gnu/curl/curl.h
libpq libpq-fe.h: /usr/include/postgresql/libpq-fe.h
services/api/config/database.yml: OK
postgresql: psql (PostgreSQL) 9.6.11
phantomjs: 1.9.8
xvfb: /usr/bin/Xvfb
graphviz: dot - graphviz version 2.38.0 (20140413.2041)
geckodriver: geckodriver 0.23.0 ( 2018-10-04)
WORKSPACE=/usr/src/arvados
Will install dependencies to /var/lib/gems
Will install arvados gems to /var/lib/arvados/test/GEMHOME/.gem/ruby/2.3.0
Gem search path is GEM_PATH=/var/lib/arvados/test/GEMHOME/.gem/ruby/2.3.0
Settings are listed in order of priority. The top value will be used.
PATH is /var/lib/arvados/test/GEMHOME/.gem/ruby/2.3.0/bin:/var/lib/gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/gems/bin:/usr/local/go/bin
Starting API, keepproxy, keep-web, ws, arv-git-httpd, and nginx ssl proxy...
Traceback (most recent call last):
  File "sdk/python/tests/run_test_server.py", line 896, in <module>
    run_keep_proxy()
  File "sdk/python/tests/run_test_server.py", line 605, in run_keep_proxy
    filters=[['service_type','=','proxy']]).execute()['items']:
  File "/var/lib/arvados/test/VENVDIR/local/lib/python2.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/var/lib/arvados/test/VENVDIR/local/lib/python2.7/site-packages/googleapiclient/http.py", line 835, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/var/lib/arvados/test/VENVDIR/local/lib/python2.7/site-packages/googleapiclient/http.py", line 162, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/src/arvados/sdk/python/arvados/api.py", line 107, in _intercept_http_request
    return self.orig_http_request(uri, method, headers=headers, **kwargs)
  File "/var/lib/arvados/test/VENVDIR/local/lib/python2.7/site-packages/httplib2/__init__.py", line 2135, in request
    cachekey,
  File "/var/lib/arvados/test/VENVDIR/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1796, in _request
    conn, request_uri, method, body, headers
  File "/var/lib/arvados/test/VENVDIR/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1737, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1108, in getresponse
    raise ResponseNotReady()
httplib.ResponseNotReady
======= test services/nodemanager -- failed to start services
Sent SIGTERM to 19469 (/usr/src/arvados/tmp/controller.pid)
Sent SIGTERM to 19355 (/usr/src/arvados/tmp/api.pid)
Leaving behind temp dirs in /var/lib/arvados/test

This used to work. (It fails the same way without --skip-install)

Restarting the docker container (arvbox restart test) works but it takes much longer.

1. services/nodemanager looks like it is just the node manager tests being their usual flaky self
2. services/api is an old problem, but would be nice to fix
3. gofmt seems legit but I don't know why it didn't come up when the code was originally committed
4. doc is legit #15043
5. services/login-sync I have no idea

Peter Amstutz wrote:

1. services/nodemanager looks like it is just the node manager tests being their usual flaky self
2. services/api is an old problem, but would be nice to fix
3. gofmt seems legit but I don't know why it didn't come up when the code was originally committed

I expect you're running a newer version of go/gofmt than Jenkins. Our gofmt checks assume all devs & CI use the same version of Go, unfortunately. We could require a specific major version in sanity checks, but I decided not to wade into that here.

1. doc is legit #15043
2. services/login-sync I have no idea

I suspect this is caused by the fact that we don't commit Gemfile.lock for services/login-sync. A new install uses mocha 1.5.0, which has deprecation warnings for mocha/mini_test, but you might have an older version. I've added '>= 1.5.0' -- with that, maybe "install services/login-sync" will fix your setup.

Restarting the docker container (arvbox restart test) works but it takes much longer

I don't know what the context is here, but it looks like api server isn't starting -- tmp/nginx_error.log might have something.

15044-test-fixes @ f4a54f243fd349995128f36b8e75c97edfbbe147

• services/login-sync works
• (from chat) the API server failure to start was happening on the second time running run-tests.sh in the same container, ie doing "exec run-test.sh" the second time, where as "restart" deletes the container and creates a new one. I could reproduce the problem yesterday, but not today. Go figure.
• still blocked on #15043 ?
• seems like we need everyone on the same page with what version of Go we're using.

If it doesn't break jenkins, LGTM