Project

General

Profile

Actions
Copy link

Story #8568

closed

[SDKs] `arv keep docker` supports Docker 1.10+

Added by Brett Smith about 9 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
SDKs
Target version:
2016-08-03 sprint
Start date:
02/26/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Description

If you try to run arv keep docker today on a host running Docker 1.10+, you'll get an exception when the tool tries to find a specific file in the generated tar file:

$ arv keep docker --project-uuid=qr1hi-j7d0g-rerv400sudof0mn name
1474M / 1474M 100.0% 
Collection saved as 'Docker image name:latest sha256:... (2)'
qr1hi-4zz18-k4b79jr1uamnila
Traceback (most recent call last):
  File "/usr/local/bin/arv-keepdocker", line 5, in <module>
    pkg_resources.run_script('arvados-python-client==0.1.20160128214108', 'arv-keepdocker')
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 517, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 1436, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/arvados_python_client-0.1.20160128214108-py2.7.egg/EGG-INFO/scripts/arv-keepdocker", line 4, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/arvados_python_client-0.1.20160128214108-py2.7.egg/arvados/commands/keepdocker.py", line 401, in main
    json_file = image_tar.extractfile(image_tar.getmember(image_hash + '/json'))
  File "/usr/lib/python2.7/tarfile.py", line 1800, in getmember
    raise KeyError("filename %r not found" % name)
KeyError: "filename 'sha256:...d2b6/json' not found"

This happens because Docker changed the image tar format in Docker 1.10. Recognize and correctly upload images generated by Docker 1.10.

Changes to make

The output of docker images --no-trunc has changed so the image hash now includes the cryptographic hash name. It looks like sha256:4fe79ae514594715386c226e89a53c8037e081603f93f65a47c82573359f70cb.

Right now arv-keepdocker looks for a JSON metadata file in the .tar under <image hash>/json:

json_file = image_tar.extractfile(image_tar.getmember(image_hash + '/json'))

This needs to be updated to look for a file named <unmarked hash>.json. So, in the example hash above, arv-keepdocker needs to look for a metadata file named 4fe79ae514594715386c226e89a53c8037e081603f93f65a47c82573359f70cb.json.

Suggest finding the metadata file with something like the following:

image_hash_type, _, raw_image_hash = image_hash.rpartition(':')
if image_hash_type:
    json_filename = raw_image_hash + '.json'
else:
    json_filename = raw_image_hash + '/json'
json_file = image_tar.extractfile(image_tar.getmember(json_filename))

I have confirmed that the JSON file still has the information arv-keepdocker needs in the new image format.

Subtasks 1 (0 open — 1 closed)

Task #9641: Review branch 8568-docker-version-supportResolvedRadhika Chippada02/26/2016

Actions

Related issues 1 (0 open — 1 closed)

Related to Arvados - Support #9628: CWL Tutorial Does not RunResolvedTom Morris07/18/2016

Actions
Actions
Copy link

Also available in: Atom PDF