Arvados

Story #8568

Updated by Brett Smith over 8 years ago

If you try to run @arv keep docker@ today on a host running Docker 1.10+, you'll get an exception when the tool tries to find a specific file in the generated tar file: 

 <pre>$ arv keep docker --project-uuid=qr1hi-j7d0g-rerv400sudof0mn name 
 1474M / 1474M 100.0%  
 Collection saved as 'Docker image name:latest sha256:... (2)' 
 qr1hi-4zz18-k4b79jr1uamnila 
 Traceback (most recent call last): 
   File "/usr/local/bin/arv-keepdocker", line 5, in <module> 
     pkg_resources.run_script('arvados-python-client==0.1.20160128214108', 'arv-keepdocker') 
   File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 517, in run_script 
     self.require(requires)[0].run_script(script_name, ns) 
   File "/usr/local/lib/python2.7/dist-packages/pkg_resources.py", line 1436, in run_script 
     exec(code, namespace, namespace) 
   File "/usr/local/lib/python2.7/dist-packages/arvados_python_client-0.1.20160128214108-py2.7.egg/EGG-INFO/scripts/arv-keepdocker", line 4, in <module> 
     main() 
   File "/usr/local/lib/python2.7/dist-packages/arvados_python_client-0.1.20160128214108-py2.7.egg/arvados/commands/keepdocker.py", line 401, in main 
     json_file = image_tar.extractfile(image_tar.getmember(image_hash + '/json')) 
   File "/usr/lib/python2.7/tarfile.py", line 1800, in getmember 
     raise KeyError("filename %r not found" % name) 
 KeyError: "filename 'sha256:...d2b6/json' not found"</pre> 

 This happens because Docker changed the image tar format in Docker 1.10.    Recognize and correctly upload images generated by Docker 1.10. 

 h2. Changes to make 

 The output of @docker images --no-trunc@ has changed so the image hash now includes the cryptographic hash name.    It looks like @sha256:4fe79ae514594715386c226e89a53c8037e081603f93f65a47c82573359f70cb@. 

 Right now arv-keepdocker looks for a JSON metadata file in the .tar under @<image hash>/json@: 

 <pre>json_file = image_tar.extractfile(image_tar.getmember(image_hash + '/json'))</pre> 

 This needs to be updated to look for a file named @<unmarked hash>.json@.    So, in the example hash above, arv-keepdocker needs to look for a metadata file named @4fe79ae514594715386c226e89a53c8037e081603f93f65a47c82573359f70cb.json@. 

 Suggest finding the metadata file with something like the following: 

 <pre><code class="python">image_hash_type, _, raw_image_hash = image_hash.rpartition(':') 
 if image_hash_type: 
     json_filename metadata_filename = raw_image_hash image_hash + '.json' 
 else: 
     json_filename metadata_filename = raw_image_hash image_hash + '/json' 
 json_file = image_tar.extractfile(image_tar.getmember(json_filename)) image_tar.extractfile(image_tar.getmember(image_hash + '/json')) 
 </code></pre> 

 I have confirmed that the JSON file still has the information arv-keepdocker needs in the new image format.

Back