Arvados

Integration with Jenkins might be conducted using example jenkinsfile:
https://docs.cypress.io/guides/guides/continuous-integration.html#Examples

Fixed Websocket Server's ExternalURL config on arvados-boot to use wss:// instead of https:// at ef109c816 - branch 16029-arvboot-wss-fix
Test run: developer-run-tests: #1774

414bb787d fixes go formatting issues. Test run: developer-run-tests-remainder: #1846

16029-arvboot-wss-fix LGTM, thanks

Updates at commit 9bd1a28a - branch 16029-cypress-testing (wb2 repo)
Test run: developer-tests-workbench2: #20 (with new workbench2-build docker image)

• Adds Cypress as an end-to-end test runner.
• Adds e2e tests for login operations (passing a token for now, when PAM login form is merged we could add those too).
• Does not rely on pre-existing fixtures, but creates users via the /auth/controller/callback private endpoint.
• Uses new arvados-server install & arvados-server boot features to have a testing cluster to test against.
• Updates Dockerfile to make a tester image with current arvados-server boot requirements. (already built on CI)

How to manually run tests?¶

• One option would be to create a docker image and use it like Jenkins does: by calling "make integration-tests" inside it. It's kind of slow because it would clone the arvados repo every time.
• Other option would be to run arvados-server boot on a certain port, also WB2 in dev mode (yarn start) and then run CYPRESS_system_token=xxx CYPRESS_controller_url=https://localhost:port yarn run cypress run

Pending¶

• Update WB2 hacking wiki to explain how to use Cypress.
• Cypress creates screenshots (on failed tests) and a video of the entire run that would be interesting to save as a job artifact, I don't know how to configure that yet.

The Dockerfile has ward@curoverse as the maintainer. It should probably be "Arvados team" (arvados@arvados.org) or Curii (info@curii.com).

The testing instructions in README.md should describe how to run tests manually, starting with building the Docker image.

The Dockerfile should go in its own subdirectory, when you do "docker build" it makes a copy of the entire workbench2 directory tree but doesn't actually access any of it.

After building the docker image, here's how I ran it (starting from the workbench2 directory):

$ docker run -ti -v$PWD:$PWD -w$PWD workbench2-build make integration-tests

There's a point where you see:

Compiled successfully!

You can now view arvados-workbench-2 in the browser.

  Local:            https://localhost:38894/
  On Your Network:  https://172.17.0.2:38894/

Note that the development build is not optimized.
To create a production build, use yarn build.

and then it waits for a rather long time without any indication something is happening. Looks like it is about two minutes:

Done in 120.13s.

It should probably print a message if it is waiting for something.

It did run the tests, but everything failed with "Timed out"

  1) Login tests shows login page on first visit:
     CypressError: Timed out retrying: expected '<div#root>' to contain 'Please log in'

Now I am suspicious that when it is waiting for Arvados services to start and is done after almost exactly 2 minutes is because it timed out and not because that's how long it took. It might need a health check before proceeding.

On re-running I notice it reinstalls Cypress along with re-cloning Arvados. We should have a way of mounting those from the outside so they don't need to be redownloaded every time.

On re-running I think I confirmed that the problem it is timing out waiting for the API server to come up, it is still compiling the sassc gem (which means API server isn't up yet) when it decided to proceed with running the tests.

If I am developing and want to re-run tests with minimum turnaround time, what is going to be the right way to do that?

Peter Amstutz wrote:

The Dockerfile has ward@curoverse as the maintainer. It should probably be "Arvados team" (arvados@arvados.org) or Curii (info@curii.com).
The testing instructions in README.md should describe how to run tests manually, starting with building the Docker image.

Ok, will do!

The Dockerfile should go in its own subdirectory, when you do "docker build" it makes a copy of the entire workbench2 directory tree but doesn't actually access any of it.

I've just started a container with this image and no volumes attached and can't find wb2 sources in it, but sure I can move the file to a subdir.

After building the docker image, here's how I ran it (starting from the workbench2 directory):
[...]
Now I am suspicious that when it is waiting for Arvados services to start and is done after almost exactly 2 minutes is because it timed out and not because that's how long it took. It might need a health check before proceeding.

The best way to run it with short turnaround times is by doing what I described on note-18, second bulletpoint on 'How to manually run tests?', the first one is the way Jenkins uses.
As we discussed on chat, arvados-boot & wb2 instances are launched as background jobs, and then waited for their URLs to be valid by using wait-on (see tools/run-integration-tests.sh script).
Arvados is waited at the discovery doc's URL to ensure both controller and railsAPI are up.

If arvados-boot fails, it will exit the process and never answer its URL so I think wait-on is correct about continuing, what I can do is making the timeout a little bigger and check for wait-on's exit status so that it doesn't try to run the tests.

On re-running I notice it reinstalls Cypress along with re-cloning Arvados. We should have a way of mounting those from the outside so they don't need to be redownloaded every time.

This is because is designed to be run by Jenkins.

On re-running I think I confirmed that the problem it is timing out waiting for the API server to come up, it is still compiling the sassc gem (which means API server isn't up yet) when it decided to proceed with running the tests.

If I am developing and want to re-run tests with minimum turnaround time, what is going to be the right way to do that?

• Manually start an arvados boot instance: go run ./cmd/arvados-server boot -type test -config ~/arvados-workbench2/tools/arvados_cluster.yml -own-temporary-database -controller-address :8000 -listen-host localhost
• Start wb2 dev instance: yarn start
• Start Cypress interactive mode: CYPRESS_system_token=xxx CYPRESS_controller_url=https://localhost:8000 yarn run cypress open

As I commented on chat, I think it would be nice to make the 'reset db' API to accept an optional param to just empty the database without loading the fixtures, wdyt?

Lucas Di Pentima wrote:

The Dockerfile should go in its own subdirectory, when you do "docker build" it makes a copy of the entire workbench2 directory tree but doesn't actually access any of it.

I've just started a container with this image and no volumes attached and can't find wb2 sources in it, but sure I can move the file to a subdir.

I mean, the way docker works is it makes a copy of the whole directory tree under where the Dockerfile sits in before building so you can use "ADD" and "COPY" to put files into the container, but if you don't do use ADD or COPY, it just wastes time/disk space making a copy you don't use and is thrown away after building the image. I don't mean it copies it into the container.

The best way to run it with short turnaround times is by doing what I described on note-18, second bulletpoint on 'How to manually run tests?', the first one is the way Jenkins uses.

Ok. At least when using Docker this should still be streamlined into a script, but that might be a different script from the jenkins one.

As we discussed on chat, arvados-boot & wb2 instances are launched as background jobs, and then waited for their URLs to be valid by using wait-on (see tools/run-integration-tests.sh script).
Arvados is waited at the discovery doc's URL to ensure both controller and railsAPI are up.

If arvados-boot fails, it will exit the process and never answer its URL so I think wait-on is correct about continuing, what I can do is making the timeout a little bigger and check for wait-on's exit status so that it doesn't try to run the tests.

That sounds right. The other thing to try is waiting on the health check aggregator which would only return a good status if when everything is up.

The easiest way I think would be:

• Manually start an arvados boot instance: go run ./cmd/arvados-server boot -type test -config ~/arvados-workbench2/tools/arvados_cluster.yml -own-temporary-database -controller-address :8000 -listen-host localhost
• Start wb2 dev instance: yarn start
• Start Cypress interactive mode: CYPRESS_system_token=xxx CYPRESS_controller_url=https://localhost:8000 yarn run cypress open

Yes, so there should a second different script to start the interactive case.

As I commented on chat, I think it would be nice to make the 'reset db' API to accept an optional param to just empty the database without loading the fixtures, wdyt?

I think we discussed this before and already agreed to do it, so yes.

Some more notes:

• As previously discussed, run-integration-tests.sh instead of "wait-for" it should use the health check aggregator endpoint to determine if the cluster is ready to run, and have a longer time out (at least 5 minutes) to accommodate packages being downloaded/installed in a fresh install

• To access an instance started by arvados-boot inside a container, from outside the container, it should include the listen-host in subjectAltName. Here's the current code:
  https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/lib/boot/cert.go#L46
  here's what arvbox does
  https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/tools/arvbox/lib/arvbox/docker/service/certificate/run#L89
  Note that the subjectAltName field distinguishes between the listen host is an IP address or hostname:
  https://dev.arvados.org/projects/arvados/repository/revisions/master/entry/tools/arvbox/lib/arvbox/docker/service/certificate/run#L51)

• It's essential to either do "REACT_APP_ARVADOS_API_HOST=... yarn start" and/or to update public/config.json, otherwise workbench2 won't know how to contact the test API server

• To run a container that is capable of running cypress in interactive mode, you need to do something like this:
  

  docker run -ti \
      -w$PWD \
      --env="DISPLAY" \
      --volume=$PWD:$PWD \
      --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
      --volume="$HOME/work/arvados:$HOME/work/arvados:rw" \
      --shm-size 2g \
      workbench2-build \
      /bin/bash
  

• To give the browser access to your screen, on your host you need to do something like:

xhost +local:`docker inspect --format='{{ .Config.Hostname }}' docker_container_name`

Tomorrow I can finally get on to reviewing the actual code of the actual tests :-)

The tests look good. I am very happy at how short they are. After struggling with the orchestration it is nice that the actual tests are easy.

I have one more comment. How hard would it be to convert the tests and support commands to Typescript? Cypress supports it:

https://docs.cypress.io/guides/tooling/typescript-support.html

Made SSL certificate's subjectAltName to include IP or DNS depending on the -listen-host argument of arvados-server boot at 7eb530d63 - branch 16029-arvboot-listenhost-ssl-altname

Peter Amstutz wrote:

The tests look good. I am very happy at how short they are. After struggling with the orchestration it is nice that the actual tests are easy.

I have one more comment. How hard would it be to convert the tests and support commands to Typescript? Cypress supports it:

https://docs.cypress.io/guides/tooling/typescript-support.html

I did tried a little bit but failed and didn't wanted to add more complexity to the task. I agree that this would be ideal, I can give it another look.

Update at b191ca173 - branch 16029-arvboot-health-externalurl Makes the health aggregator accessible from the outside of arvados boot. This is to make the integration test script able to ask for its port by accessing the exported config endpoint.

Branch on previous comment isn't needed as arvados boot returns its URL when all the health checks are OK.
Updated orchestration script to follow the pattern of https://github.com/arvados/arvados/blob/master/lib/boot/example.sh at commit 855193b8 (wb2 repo)
Test run: developer-tests-workbench2: #21

Update at commit 0c1be947 (wb2 repo)
Test run: developer-tests-workbench2: #22

• Adds options to the orchestration script to allow interactive mode (useful for developing)

Usage: run-integration-tests.sh [options]
Options:
  -i            Run Cypress in interactive mode.
  -a PATH       Arvados dir. If PATH doesn't exist, a repo clone is downloaded there.
  -w PATH       Workbench2 dir. Default: Current working directory

Awesome. This looks good to me. I pushed a couple of minor tweaks to 16029-cypress-testing -- now f3569ce613f0e19a63149c08be68e0bcbb7eaf92

Please merge and let's get it working on Jenkins.