Arvados

During standup we agreed (Peter/Tom/Lucas/me) that this ticket has not aged well since we don't support py2 anymore. Migrating all python tooling to py3 is the best idea we have.

apt-get  remove $(COLUMNS=160 dpkg -l | grep -E 'python[^3]'| awk '{ print $2 }' | grep -v 'dh-python' )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  cython libpython-dev libpython-stdlib libpython2.7 libpython2.7-dev libpython2.7-minimal
  libpython2.7-stdlib linkchecker python python-chardet python-epydoc python-minimal
  python-pip python-pip-whl python-pkg-resources python-pycurl python-requests
  python-setuptools python-six python-urllib3 python-virtualenv python-yaml python2.7
  python2.7-minimal python3-pip python3-virtualenv virtualenv
0 upgraded, 0 newly installed, 27 to remove and 1 not upgraded.
After this operation, 79.7 MB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 38304 files and directories currently installed.)
Removing cython (0.25.2-1) ...
Removing libpython-dev:amd64 (2.7.13-2) ...
Removing python-yaml (3.12-1) ...
Removing linkchecker (9.3-4) ...
Removing python-requests (2.12.4-1) ...
Removing python-urllib3 (1.19.1-1) ...
Removing python-six (1.10.0-3) ...
Removing python-virtualenv (15.1.0+ds-1) ...
Removing python-setuptools (33.1.1-1) ...
Removing libpython2.7-dev:amd64 (2.7.13-2+deb9u3) ...
Removing libpython2.7:amd64 (2.7.13-2+deb9u3) ...
Removing python-chardet (2.3.0-2) ...
Removing python-epydoc (3.0.1+dfsg-14) ...
Removing python-pip (9.0.1-2+deb9u1) ...
Removing virtualenv (15.1.0+ds-1) ...
Removing python3-virtualenv (15.1.0+ds-1) ...
Removing python-pkg-resources (33.1.1-1) ...
Removing python-pycurl (7.43.0-2) ...
Removing python3-pip (9.0.1-2+deb9u1) ...
Removing python (2.7.13-2) ...
Removing libpython-stdlib:amd64 (2.7.13-2) ...
Removing python2.7 (2.7.13-2+deb9u3) ...
Removing libpython2.7-stdlib:amd64 (2.7.13-2+deb9u3) ...
Removing python-minimal (2.7.13-2) ...
Removing python2.7-minimal (2.7.13-2+deb9u3) ...
Removing libpython2.7-minimal:amd64 (2.7.13-2+deb9u3) ...
Removing python-pip-whl (9.0.1-2+deb9u1) ...
Processing triggers for mime-support (3.60) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
root@4e513ebf867f:/usr/src/arvados# apt-get install python3-pip python3-virtualenv
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  dbus python-pip-whl python3-cffi-backend python3-crypto python3-cryptography python3-dbus
  python3-gi python3-idna python3-keyring python3-keyrings.alt python3-pyasn1
  python3-secretstorage python3-six python3-wheel python3-xdg
Suggested packages:
  default-dbus-session-bus | dbus-session-bus python3-crypto-dbg python-crypto-doc
  python-cryptography-doc python3-cryptography-vectors python-dbus-doc python3-dbus-dbg
  gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0 python3-pykde4 doc-base
  python-secretstorage-doc
The following NEW packages will be installed:
  dbus python-pip-whl python3-cffi-backend python3-crypto python3-cryptography python3-dbus
  python3-gi python3-idna python3-keyring python3-keyrings.alt python3-pip python3-pyasn1
  python3-secretstorage python3-six python3-virtualenv python3-wheel python3-xdg
0 upgraded, 17 newly installed, 0 to remove and 1 not upgraded.
Need to get 3,245 kB of archives.
After this operation, 8,668 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://deb.debian.org/debian stretch/main amd64 dbus amd64 1.10.32-0+deb9u1 [213 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 python-pip-whl all 9.0.1-2+deb9u1 [1,399 kB]
Get:3 http://deb.debian.org/debian stretch/main amd64 python3-cffi-backend amd64 1.9.1-2 [70.1 kB]
Get:4 http://deb.debian.org/debian stretch/main amd64 python3-crypto amd64 2.6.1-7 [259 kB]
Get:5 http://deb.debian.org/debian stretch/main amd64 python3-idna all 2.2-1 [32.7 kB]
Get:6 http://deb.debian.org/debian stretch/main amd64 python3-pyasn1 all 0.1.9-2 [34.5 kB]
Get:7 http://deb.debian.org/debian stretch/main amd64 python3-six all 1.10.0-3 [14.4 kB]
Get:8 http://deb.debian.org/debian stretch/main amd64 python3-cryptography amd64 1.7.1-3+deb9u2 [211 kB]
Get:9 http://deb.debian.org/debian stretch/main amd64 python3-dbus amd64 1.2.4-1+b1 [184 kB]
Get:10 http://deb.debian.org/debian stretch/main amd64 python3-gi amd64 3.22.0-2 [473 kB]
Get:11 http://deb.debian.org/debian stretch/main amd64 python3-secretstorage all 2.3.1-2 [14.2 kB]
Get:12 http://deb.debian.org/debian stretch/main amd64 python3-keyring all 10.1-1 [36.8 kB]
Get:13 http://deb.debian.org/debian stretch/main amd64 python3-keyrings.alt all 1.3-1 [16.2 kB]
Get:14 http://deb.debian.org/debian stretch/main amd64 python3-pip all 9.0.1-2+deb9u1 [142 kB]
Get:15 http://deb.debian.org/debian stretch/main amd64 python3-virtualenv all 15.1.0+ds-1 [57.7 kB]
Get:16 http://deb.debian.org/debian stretch/main amd64 python3-wheel all 0.29.0-2 [51.8 kB]
Get:17 http://deb.debian.org/debian stretch/main amd64 python3-xdg all 0.25-4 [35.7 kB]
Fetched 3,245 kB in 1s (1,949 kB/s)     
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package dbus.
(Reading database ... 36214 files and directories currently installed.)
Preparing to unpack .../00-dbus_1.10.32-0+deb9u1_amd64.deb ...
Unpacking dbus (1.10.32-0+deb9u1) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../01-python-pip-whl_9.0.1-2+deb9u1_all.deb ...
Unpacking python-pip-whl (9.0.1-2+deb9u1) ...
Selecting previously unselected package python3-cffi-backend.
Preparing to unpack .../02-python3-cffi-backend_1.9.1-2_amd64.deb ...
Unpacking python3-cffi-backend (1.9.1-2) ...
Selecting previously unselected package python3-crypto.
Preparing to unpack .../03-python3-crypto_2.6.1-7_amd64.deb ...
Unpacking python3-crypto (2.6.1-7) ...
Selecting previously unselected package python3-idna.
Preparing to unpack .../04-python3-idna_2.2-1_all.deb ...
Unpacking python3-idna (2.2-1) ...
Selecting previously unselected package python3-pyasn1.
Preparing to unpack .../05-python3-pyasn1_0.1.9-2_all.deb ...
Unpacking python3-pyasn1 (0.1.9-2) ...
Selecting previously unselected package python3-six.
Preparing to unpack .../06-python3-six_1.10.0-3_all.deb ...
Unpacking python3-six (1.10.0-3) ...
Selecting previously unselected package python3-cryptography.
Preparing to unpack .../07-python3-cryptography_1.7.1-3+deb9u2_amd64.deb ...
Unpacking python3-cryptography (1.7.1-3+deb9u2) ...
Selecting previously unselected package python3-dbus.
Preparing to unpack .../08-python3-dbus_1.2.4-1+b1_amd64.deb ...
Unpacking python3-dbus (1.2.4-1+b1) ...
Selecting previously unselected package python3-gi.
Preparing to unpack .../09-python3-gi_3.22.0-2_amd64.deb ...
Unpacking python3-gi (3.22.0-2) ...
Selecting previously unselected package python3-secretstorage.
Preparing to unpack .../10-python3-secretstorage_2.3.1-2_all.deb ...
Unpacking python3-secretstorage (2.3.1-2) ...
Selecting previously unselected package python3-keyring.
Preparing to unpack .../11-python3-keyring_10.1-1_all.deb ...
Unpacking python3-keyring (10.1-1) ...
Selecting previously unselected package python3-keyrings.alt.
Preparing to unpack .../12-python3-keyrings.alt_1.3-1_all.deb ...
Unpacking python3-keyrings.alt (1.3-1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../13-python3-pip_9.0.1-2+deb9u1_all.deb ...
Unpacking python3-pip (9.0.1-2+deb9u1) ...
Selecting previously unselected package python3-virtualenv.
Preparing to unpack .../14-python3-virtualenv_15.1.0+ds-1_all.deb ...
Unpacking python3-virtualenv (15.1.0+ds-1) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../15-python3-wheel_0.29.0-2_all.deb ...
Unpacking python3-wheel (0.29.0-2) ...
Selecting previously unselected package python3-xdg.
Preparing to unpack .../16-python3-xdg_0.25-4_all.deb ...
Unpacking python3-xdg (0.25-4) ...
Setting up python-pip-whl (9.0.1-2+deb9u1) ...
Setting up python3-dbus (1.2.4-1+b1) ...
Setting up python3-cffi-backend (1.9.1-2) ...
Setting up python3-crypto (2.6.1-7) ...
Setting up python3-idna (2.2-1) ...
Setting up python3-xdg (0.25-4) ...
Setting up python3-keyrings.alt (1.3-1) ...
Setting up python3-six (1.10.0-3) ...
Setting up python3-wheel (0.29.0-2) ...
Setting up python3-gi (3.22.0-2) ...
Setting up python3-pyasn1 (0.1.9-2) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Setting up python3-virtualenv (15.1.0+ds-1) ...
Setting up python3-pip (9.0.1-2+deb9u1) ...
Setting up dbus (1.10.32-0+deb9u1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up python3-cryptography (1.7.1-3+deb9u2) ...
Setting up python3-secretstorage (2.3.1-2) ...
Setting up python3-keyring (10.1-1) ...

Intalled python-venv

this will allow to do

python3 -m venv /DIR

python3-httplib2 is needed

++ python3 sdk/python/tests/run_test_server.py setup_config
Traceback (most recent call last):
  File "sdk/python/tests/run_test_server.py", line 13, in <module>
    import httplib2
ImportError: No module named 'httplib2'
+ eval

62b3435fc45adff01541c508be22e10be83427ce

There are still some issues with importing arvados in the py3 environment

TestArvKeepGet#test_help = Traceback (most recent call last):
  File "./bin/arv-put", line 6, in <module>
    from arvados.commands.put import main
ImportError: No module named 'arvados'
Traceback (most recent call last):
  File "./bin/arv-put", line 6, in <module>
    from arvados.commands.put import main
ImportError: No module named 'arvados'
Traceback (most recent call last):
  File "./bin/arv-put", line 6, in <module>
    from arvados.commands.put import main
ImportError: No module named 'arvados'
0.11 s = F

Before merging into master, will need to remove the pycurl pin added here: https://dev.arvados.org/issues/16795#note-3

Almost there, but not 100%:

developer-run-tests: #2066

If we use the python2.7 "own python" from package this is the behaviour:

nico@shell:~$ arv-get --version
/usr/bin/arv-get 2.0.3
nico@shell:~$ head --lines=1 /usr/bin/arv-get
#!/usr/share/python2.7/dist/python-arvados-python-client/bin/python
nico@shell:~$ arv-get --version 2> /dev/null
nico@shell:~$ 

1) Failure:
TestArvKeepGet#test_get_version [/tmp/workspace/developer-run-tests-remainder/sdk/cli/test/test_arv-keep-get.rb:31]:
STDOUT not expected: './bin/arv-get 2.1.0.dev20200826203407
'.
Expected "./bin/arv-get 2.1.0.dev20200826203407\n" to be empty.

this is the test from sdk/cli/test/test_arv-keep-get.rb

  def test_get_version
    out, err = capture_subprocess_io do
      assert_arv_get '--version'
    end
    assert_empty(out, "STDOUT not expected: '#{out}'")
    assert_match(/[0-9]+\.[0-9]+\.[0-9]+/, err, "Version information incorrect: '#{err}'")
  end

nico@shell:~$ python3 /usr/bin/arv-get --version
Traceback (most recent call last):
  File "/usr/bin/arv-get", line 8, in <module>
    from arvados.commands.get import main
ModuleNotFoundError: No module named 'arvados'

because of the lack of that library installed:

nico@shell:~$ dpkg -l | grep python3-arvados-python-client 
nico@shell:~$ 

So here we have to make sure that when we change our packages to have the right dependency when deploying

after installing:

nico@shell:~$ dpkg -l | grep python3-arvados-python-client 
ii  python3-arvados-python-client    2.0.4-1                      amd64        Arvados client library
nico@shell:~$ head /usr/bin/arv-get
#!/usr/share/python3/dist/python3-arvados-python-client/bin/python
nico@shell:~$ /usr/share/python3/dist/python3-arvados-python-client/bin/python /usr/bin/arv-get --version 2> /dev/null
/usr/bin/arv-get 2.0.4

https://bugs.python.org/issue18920

developer-run-tests: #2071 successfull run!

A few comments:

• shouldn't commit the go.sum and go.mod changes

noted

• trailing space added in run-tests.sh line 657

• run-tests.sh:

@@ -558,7 +555,7 @@ setup_ruby_environment() {
                 fi
             done
             "$bundle" version | tee /dev/stderr | grep -q 'version 2'
-        ) || fatal 'install bundler'
+        ) #|| fatal 'install bundler' ## This is needed for now.
     fi
 }

This seems like it should not be part of the patch?

• The patch is adding a few "FIXME" lines, do we need additional changes there?

• Any reason you're removing the '-x' from the bash invocation (2 places)?

Did most of the changes.

about $pythonstuff...

install_all tries to do a pip install of the packages in $pythonstuff. as of 5aca41295

all of them have the suffix ":py3"

pythonstuff=(
    sdk/python:py3
    sdk/cwl:py3
    services/dockercleaner:py3
    services/fuse:py3
    tools/crunchstat-summary:py3
)

and I've seen some code that tries to remove the ones ending in ":py3"[1] by doing:

    for p in "${pythonstuff[@]}" 
    do
            dir=${p%:py3}

so I think we should remove this part.

My proposal is 4cfab58ae and running developer-run-tests: #2073 now

[1] man bash
${parameter%word}
${parameter%%word}
    Remove matching suffix pattern. The word is expanded to produce a pattern just as in pathname expansion. If the pattern matches a trailing portion of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern (the ''%'' case) or the longest matching pattern (the ''%%'' case) deleted. If parameter is @ or *, the pattern removal operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with @ or *, the pattern removal operation is applied to each member of the array in turn, and the expansion is the resultant list. 

Nico César wrote:

Did most of the changes.

about $pythonstuff...

install_all tries to do a pip install of the packages in $pythonstuff. as of 5aca41295

all of them have the suffix ":py3"
[...]

and I've seen some code that tries to remove the ones ending in ":py3"[1] by doing:
[...]

so I think we should remove this part.

My proposal is 4cfab58ae and running developer-run-tests: #2073 now

[...]

Tests passed, LGTM. Just make sure after you merge master into your branch that you then change this line in sdk/python/setup.py:

'pycurl >=7.19.5.1, <7.43.0.4', # 7.43.0.4 removes support for python2

to remove the '<7.43.0.4' part, which should no longer be needed after you purged python2. It was introduced in master after you forked.

Could also do that as a separate commit, of course.

developer-run-tests: #2073 passed and ward said LGTM.

I'll go ahead and merge it

run-tests: #3964

run-tests: #3965

Another remnant in arados-server, cf. 94a7479527652cd513a35f75e1da1798d9fdc213 on branch 15888-remove-more-python2-remnants.

94a7479527652cd513a35f75e1da1798d9fdc213 LGTM,

I noticed developer-run-tests: #2083
with commit 94a7479527652cd513a35f75e1da1798d9fdc213, but it seems to be a flaky test.

Nico César wrote:

94a7479527652cd513a35f75e1da1798d9fdc213 LGTM,

I noticed developer-run-tests: #2083
with commit 94a7479527652cd513a35f75e1da1798d9fdc213, but it seems to be a flaky test.

Thanks, merged it. It's a flaky test indeed.