Arvados

22428-run-tests-targets @ 5c280660c651e036333d8fce5c358ba9792c5a56 - developer-run-tests: #4588

• All agreed upon points are implemented / addressed.
  • Yes
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • N/A
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • Since this is test scaffolding, you can run the commands above to see that the results are now consistent and complete.
• Documentation has been updated.
  • This kind of is a documentation change.
• Behaves appropriately at the intended scale (describe intended scale).
  • No change in scale.
• Considered backwards and forwards compatibility issues between client and server.
  • No change in compatibility, this is just for human readers.
• Follows our coding standards and GUI style guidelines.
  • N/A (no shell style)

• As a first comment: I like a lot the new target list formatting.
• What I understand from the ticket description, running run-tests.sh --help should give a different target list from running run-tests.sh --interactive --help. This is not happening on my end. I probably misunderstood the description, can you confirm?
• When running run-tests.sh --help from main I get a list of 62 targets, while when I do the same on this branch, I get just 20 targets.
• Some of the targets on the new version are suspicious and might not be valid
  • all
  • arvados_version.py
  • go
  • gofmt
• When running --help on the new version, I'm getting an error message before the help message: dirname: missing operand
• This is being tried with GNU bash, version 5.1.4(1)-release (aarch64-unknown-linux-gnu)

Example of what I'm seeing (except for the target formatting; here's being rendered differently):

$ ./run-tests.sh --help
dirname: missing operand
Try 'dirname --help' for more information.
run-tests.sh: Install and test Arvados components.

Exit non-zero if any tests fail.

Syntax:
        run-tests.sh WORKSPACE=/path/to/arvados [options]

Options:

--skip FOO     Do not test the FOO component.
--skip sanity  Skip initial dev environment sanity checks.
--skip install Do not run any install steps. Just run tests.
               You should provide GOPATH, GEMHOME, and VENVDIR options
               from a previous invocation if you use this option.
--only FOO     Do not test anything except the FOO component. If given
               more than once, all specified test suites are run.
--temp DIR     Install components and dependencies under DIR instead of
               making a new temporary directory. Implies --leave-temp.
--leave-temp   Do not remove GOPATH, virtualenv, and other temp dirs at exit.
               Instead, show the path to give as --temp to reuse them in
               subsequent invocations.
--repeat N     Repeat each install/test step until it succeeds N times.
--retry        Prompt to retry if an install or test suite fails.
--only-install Run specific install step. If given more than once,
               all but the last are ignored.
--short        Skip (or scale down) some slow tests.
--interactive  Set up, then prompt for test/install steps to perform.
WORKSPACE=path Arvados source tree to test.
CONFIGSRC=path Dir with config.yml file containing PostgreSQL section for use by tests.
services/api_test="TEST=test/functional/arvados/v1/collections_controller_test.rb" 
               Restrict apiserver tests to the given file
sdk/python_test="tests/test_api.py::ArvadosApiTest" 
               Restrict Python SDK tests to the given class
lib/dispatchcloud_test="-check.vv" 
               Show all log messages, even when tests pass (also works
               with services/keepstore_test etc.)
ARVADOS_DEBUG=1
               Print more debug messages
envvar=value   Set $envvar to value. Primarily useful for WORKSPACE,
               *_test, and other examples shown above.

Assuming "--skip install" is not given, all components are installed
into $GOPATH, $VENDIR, and $GEMHOME before running any tests. Many
test suites depend on other components being installed, and installing
everything tends to be quicker than debugging dependencies.

As a special concession to the current CI server config, CONFIGSRC
defaults to /home/lucas/arvados-api-server if that directory exists.

More information and background:

https://dev.arvados.org/projects/arvados/wiki/Running_tests

Available targets:

all                        sdk/java-v2                    services/fuse
arvados_version.py        sdk/python                    services/login-sync
doc                        sdk/R                        services/workbench2_integration
go                        sdk/ruby                        services/workbench2_units
gofmt                    sdk/ruby-google-api-client        tools/cluster-activity
sdk/cli                    services/api                    tools/crunchstat-summary
sdk/cwl                    services/dockercleaner
$ echo $?
1

Lucas Di Pentima wrote in #note-4:

• What I understand from the ticket description, running run-tests.sh --help should give a different target list from running run-tests.sh --interactive --help. This is not happening on my end. I probably misunderstood the description, can you confirm?

You can get a target list two different ways:

1. Run run-tests.sh --help
2. Run run-tests.sh --interactive, then at the "What next?" prompt, say help

In main, those lists are different. That is a bug; they should be the same. This branch fixes that bug.

• When running run-tests.sh --help from main I get a list of 62 targets, while when I do the same on this branch, I get just 20 targets.
• When running --help on the new version, I'm getting an error message before the help message: dirname: missing operand

It sounds like you are running run-tests.sh without defining WORKSPACE pointed at your arvados checkout. I have added some code to guard against this and avoid printing any target list at all when this is the case. However, in general, run-tests.sh assumes WORKSPACE is defined correctly, and there are about fifty billion ways it can break if that's not true, and fixing them all is out of scope for this branch.

• Some of the targets on the new version are suspicious and might not be valid
  • all
  • arvados_version.py
  • go
  • gofmt

These are in fact all valid. You can find all the corresponding test_TARGET functions for all of these targets in run-tests.sh.

Now at a97b50ec6d9f8cd088351f682361459076bb9d80 - developer-run-tests: #4596

LGTM, thanks!