Arvados

Generally anything which is an internal tool or isn't critical doesn't need to be ported to Python 3 until it's ever needed again.

For all other components, the process should be: 1) add Python 3 support, 2) make Python 3 the default, 3) remove Python 2 support, but we don't want to do 2 & 3 simultaneously so that we have at least some period of time (6 months?) where both are supported to make the lives of our customers who are migrating a little easier. Discontinuities are difficult to deal with.

Some comments on the various questions:

• /sdk/cwl - Do we need to continue supporting py2? If not, we can simplify /build/build-dev-docker-jobs-image.sh and /sdk/dev-jobs.dockerfile.

No need to support Python 2 on an ongoing basis, but we should allow for a little overlap with both Python 2 & 3 support. Perhaps look at removing Python 2 in 15.2 timeframe in 6 months or so (Summer 2020)

Supports py3, but do we test it anywhere:

• /tools/keep-xref/keep-xref.py

Test coverage is orthogonal. This was a one-time script, so not a huge issue for test coverage.

Currently supports py2 only. Decide if we want to update:

• /doc/gen_api_method_docs.py
• /doc/gen_api_schema_docs.py

Just used to generate the API doc skeleton, so only needed if the API changes. Python 3 porting can be deferred until then.

• /services/keep/tools/traffic_test.py

Benchmarking script from 2014. It can be updated when/if it's ever used again.

Needs to be tested with py3 and potentially updated:

• /services/arv-web/sample-wsgi-app/passenger_wsgi.py
• /services/arv-web/arv-web.py

Does anyone use this? Since it's an example, I think we can defer upgrading it until someone asks.

Arvbox:

• yml_override.py - Test against py3.
• Dockerfile.base - Default to py3 and remove py2 installation.

Don't remove Python 2 immediately. See above.

CI and testing:

• Decide on a py3 version. Is there a need to test against multiple versions of py3?

Perhaps eventually, but we can consider that separately. Ideally we'd test at least the currently specified minimum version that we require (to make sure that what we say we support works) and the latest Python release (to get early visibility into things that might break in the future).

• Run run_test_server.py and all python tests using py3.

Aren't we already testing against both Python 3 and Python 2?

• Outline process for updating python to simplify future upgrades.

Is this specific to Jenkins? Is it more than just running apt-get? If it's really a thing, let's create a ticket for it with a description of what needs to be done.

• arvados-cwl-runner is packaged with its own python version. Can this and subsequently /docker/jobs/Dockerfile be updated?

The CLI tools are bundled with their own Python versions as well, I believe.

Docs:

• Update python cookbook examples and any direct references to py2.

These are mostly code snippets that are aren't too involved, so should be quick to review (and hopefully won't require any changes).

• /sdk/pam - Doesn't support py3 and we don't plan on updating

• #14964 "port arvados-pam to Python 3"
• #15348 "PAM module in Go"

• /services/keep/tools/traffic_test.py

This script should just be deleted -- we have keep-exercise now.

Needs to be tested with py3 and potentially updated:

• /services/arv-web/sample-wsgi-app/passenger_wsgi.py
• /services/arv-web/arv-web.py

I suppose we should either update these, or stop advertising them at https://doc.arvados.org/user/topics/arv-web.html

• Decide on a py3 version. Is there a need to test against multiple versions of py3?

Would be good, but sounds out-of-scope.

• Run run_test_server.py and all python tests using py3.

I think changing VENVDIR/bin/activate to VENV3DIR/bin/activate in run-tests.sh would go a long way. We already call run_test_server.py from Python3 when testing sdk/python:py3, so it's at least partly Python3-compatible.

• /build/run_tests.sh - Remove support for both python versions.
• /build/run_library.sh - Remove support for both python versions.

Perhaps first remove the dependency on Python2, but still run the Python2 tests if it's installed? We should help people get off Python2 for existing reasons, but we should probably avoid creating additional reasons.

• arvados-cwl-runner is packaged with its own python version. Can this and subsequently /docker/jobs/Dockerfile be updated?

I think if a-c-r works in Python3 then packages & jobs image should use Python3.

Unless there are supported distros where a-c-r runs correctly on Python2, but the version of Python3 is too old/incompatible...?

Summary of the above comments, meant to describe engineering and ops tasks that can be broken out into tickets.

Default to py3: Epic located here: #14532

• #15886 Update the following or stop advertising them at https://doc.arvados.org/user/topics/arv-web.html
  • /services/arv-web/sample-wsgi-app/passenger_wsgi.py
  • /services/arv-web/arv-web.py
• #15887 Update python cookbook examples and any direct references to py2 in the docs.
• #15888 Change VENVDIR/bin/activate to VENV3DIR/bin/activate in run-tests.sh. If py2 isn't installed, run-tests.sh should succeed (by skipping the py2 tests). If py2 is installed, run-tests.sh should not skip the py2 tests.
• #15889 Delete /services/keep/tools/traffic_test.py, we have keep-exercise now.

Remove py2 support: (6 months ~ Summer 2020) Epic located here: #15890

• #15891 sdk/cwl - simplify /build/build-dev-docker-jobs-image.sh and /sdk/dev-jobs.dockerfile
• #15892 Arvbox
  • Dockerfile.base - Remove py2 installation.
  • common.sh - Simplify pip installation.
• #15893 Update build scripts:
  • /build/run_tests.sh - Remove support for py2.
  • /build/run_library.sh - Remove support for py2.

• Stop building py2 packages
• Rename py3 packages to the default name.
• Update anything that is packaged with its own version of python (arvados-cwl-runner and other CLI tools). Consider supported distros where a-c-r runs correctly on Python2, but the distro version of Python3 is too old/incompatible.
• Remove py2 from /docker/jobs/Dockerfile.
• Update build/package-build-dockerfiles to remove py2.

Notes:

• Only update the following to py3 if we need to change them for any reason
  • /doc/gen_api_method_docs.py
  • /doc/gen_api_schema_docs.py
• sdk/pam - Doesn't support py3 and we don't plan on updating. See #14964 and #15348 for the Go port.

• Eventually, test against multiple versions of py3:
1. The specified minimum version that we require
2. The latest Python release (to get early visibility into things that might break in the future).