Arvados

Bug #17390

Updated by Peter Amstutz over 3 years ago

Goals of this ticket: 

 * to have all the expected behaviour for arvados-cwl-runner 
 * to add to the documentation this behaviour (if needed) 
 * to add necesary tests to make sure we comply with this behaviour (if needed) 

 h2. Current command line arguments (arvados 2.1.0) : 

 <pre> 
 usage: arvados-cwl-runner [-h] [--basedir BASEDIR] [--outdir OUTDIR] 
                           [--eval-timeout EVAL_TIMEOUT] 
                           [--print-dot | --version | --validate] 
                           [--verbose | --quiet | --debug] [--metrics] 
                           [--tool-help] [--enable-reuse | --disable-reuse] 
                           [--project-uuid UUID] [--output-name OUTPUT_NAME] 
                           [--output-tags OUTPUT_TAGS] 
                           [--ignore-docker-for-reuse] 
                           [--submit | --local | --create-template | --create-workflow | --update-workflow UUID] 
                           [--wait | --no-wait] 
                           [--log-timestamps | --no-log-timestamps] 
                           [--api {containers}] [--compute-checksum] 
                           [--submit-runner-ram SUBMIT_RUNNER_RAM] 
                           [--submit-runner-image SUBMIT_RUNNER_IMAGE] 
                           [--always-submit-runner] 
                           [--submit-request-uuid UUID | --submit-runner-cluster CLUSTER_ID] 
                           [--collection-cache-size COLLECTION_CACHE_SIZE] 
                           [--name NAME] [--on-error {stop,continue}] 
                           [--enable-dev] [--storage-classes STORAGE_CLASSES] 
                           [--intermediate-output-ttl N] [--priority PRIORITY] 
                           [--thread-count THREAD_COUNT] 
                           [--http-timeout HTTP_TIMEOUT] [--skip-schemas] 
                           [--trash-intermediate | --no-trash-intermediate] 
                           workflow ... 
 (..) 
   --storage-classes STORAGE_CLASSES 
                         Specify comma separated list of storage classes to be 
                         used when saving workflow output to Keep. 
 </pre> 


 



 h3. base casecase 

 <pre> 
 arvados-cwl-runner --name "CWL diagnostics hasher" --create-workflow hasher-workflow.cwl hasher-workflow-job.yml --storage-classes STORAGE_CLASSES 
 </pre>  

 Existing behavior is final output collection goes to the storage class. 

 Currently all intermediates go to 'default'. 

 Expected behaviour:  

 * Need to be able to set the storage class of output of a container request 
 * Should be able to choose a storage class and have it used for intermediate results through the workflow 
 * Need to be able to specify the storage class for a given workflow step ??? 

 In CWL: 

 <pre> 
 StorageClassHint: 
   storageClasses: [foo, bar] 
 </pre> 

 Turns this into "output_storage_classes" on the container request. 

 Command line parameter applies to everything. 

 Want to support a case where all the intermediates go to fast storage and only the final outputs go into long term storage. 

Back