Arvados

cwl-runner allows input arguments to be specified on the command line rather than in an inputs file.

arvados-cwl-runner appears to allow this as well:

$ arvados-cwl-runner bwa-mem.cwl --help
2017-12-28 01:29:48 cwltool INFO: /usr/bin/arvados-cwl-runner dc78526ba494973df7d298825e20503353e92adf 1.0.20171116210428, arvados-python-client 0.1.20171109204045, cwltool 1.0.20170928192020
2017-12-28 01:29:48 cwltool INFO: Resolved 'bwa-mem.cwl' to 'file:///tmp/arvados/doc/user/cwl/bwa-mem/bwa-mem.cwl'
usage: bwa-mem.cwl [-h] --PL PL --group_id GROUP_ID --read_p1 READ_P1
                   [--read_p2 READ_P2] --reference REFERENCE --sample_id
                   SAMPLE_ID
                   [job_order]

positional arguments:
  job_order             Job input json file

optional arguments:
  -h, --help            show this help message and exit
  --PL PL
  --group_id GROUP_ID
  --read_p1 READ_P1     The reads, in fastq format.
  --read_p2 READ_P2     For mate paired reads, the second file (optional).
  --reference REFERENCE
                        The index files produced by `bwa index`
  --sample_id SAMPLE_ID

My expectation would be that I could specify the inputs to this workflow using the `--read_p1`, `--read_p2`, and `--reference` arguments using a keep locator. However, when I try this, I get an error:

$ arvados-cwl-runner bwa-mem.cwl --reference keep:2463fa9efeb75e099685528b3b9071e0+438/19.fasta.bwt --read_p1 keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_1.fastq --read_p2 keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_2.fastq --group_id arvados_tutorial --sample_id HWI-ST1027_129 --PL illumina
2017-12-28 01:31:25 cwltool INFO: /usr/bin/arvados-cwl-runner dc78526ba494973df7d298825e20503353e92adf 1.0.20171116210428, arvados-python-client 0.1.20171109204045, cwltool 1.0.20170928192020
2017-12-28 01:31:25 cwltool INFO: Resolved 'bwa-mem.cwl' to 'file:///tmp/arvados/doc/user/cwl/bwa-mem/bwa-mem.cwl'
2017-12-28 01:31:26 cwltool ERROR: Unhandled error, try again with --debug for more information:
  [Errno 2] No such file or directory: '/tmp/arvados/doc/user/cwl/bwa-mem/keep:2463fa9efeb75e099685528b3b9071e0+438/19.fasta.bwt'

The same with `--debug`:

$ arvados-cwl-runner --debug bwa-mem.cwl --reference keep:2463fa9efeb75e099685528b3b9071e0+438/19.fasta.bwt --read_p1 keep:ae480c5099
b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_1.fastq --read_p2 keep:ae480c5099b81e17267b7445e35b4bc7+180/HWI-ST1027_129_D0THKACXX.1_2.fastq --group_id arvados_tutorial --sample_id HWI-S
T1027_129 --PL illumina
2017-12-28 01:31:55 cwltool INFO: /usr/bin/arvados-cwl-runner dc78526ba494973df7d298825e20503353e92adf 1.0.20171116210428, arvados-python-client 0.1.20171109204045, cwltool 1.0.20170928192020
2017-12-28 01:31:55 cwltool INFO: Resolved 'bwa-mem.cwl' to 'file:///tmp/arvados/doc/user/cwl/bwa-mem/bwa-mem.cwl'
2017-12-28 01:31:55 cwltool DEBUG: Parsed job order from command line: {
    "read_p2": {
        "class": "File",
        "location": "file:///tmp/arvados/doc/user/cwl/bwa-mem/keep%3Aae480c5099b81e17267b7445e35b4bc7%2B180/HWI-ST1027_129_D0THKACXX.1_2.fastq" 
    },
    "read_p1": {
        "class": "File",
        "location": "file:///tmp/arvados/doc/user/cwl/bwa-mem/keep%3Aae480c5099b81e17267b7445e35b4bc7%2B180/HWI-ST1027_129_D0THKACXX.1_1.fastq" 
    },
    "reference": {
        "class": "File",
        "location": "file:///tmp/arvados/doc/user/cwl/bwa-mem/keep%3A2463fa9efeb75e099685528b3b9071e0%2B438/19.fasta.bwt" 
    },
    "sample_id": "HWI-ST1027_129",
    "group_id": "arvados_tutorial",
    "id": "bwa-mem.cwl",
    "PL": "illumina" 
}
2017-12-28 01:31:55 cwltool ERROR: Unhandled error, try again with --debug for more information:
  [Errno 2] No such file or directory: '/tmp/arvados/doc/user/cwl/bwa-mem/keep:2463fa9efeb75e099685528b3b9071e0+438/19.fasta.bwt'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cwltool/main.py", line 913, in main
    **vars(args))
  File "/usr/lib/python2.7/dist-packages/arvados_cwl/__init__.py", line 384, in arv_executor
    tool, job_order)
  File "/usr/lib/python2.7/dist-packages/arvados_cwl/runner.py", line 217, in upload_job_order
    False)
  File "/usr/lib/python2.7/dist-packages/arvados_cwl/runner.py", line 141, in upload_dependencies
    single_collection=True)
  File "/usr/lib/python2.7/dist-packages/arvados_cwl/pathmapper.py", line 51, in __init__
    super(ArvPathMapper, self).__init__(referenced_files, input_basedir, None)
  File "/usr/lib/python2.7/dist-packages/cwltool/pathmapper.py", line 229, in __init__
    self.setup(dedup(referenced_files), basedir)
  File "/usr/lib/python2.7/dist-packages/arvados_cwl/pathmapper.py", line 137, in setup
    self.visit(srcobj, uploadfiles)
  File "/usr/lib/python2.7/dist-packages/arvados_cwl/pathmapper.py", line 71, in visit
    raiseOSError=True)
  File "/usr/lib/python2.7/dist-packages/arvados/commands/run.py", line 118, in statfile
    st = os.stat(absfn)
OSError: [Errno 2] No such file or directory: '/tmp/arvados/doc/user/cwl/bwa-mem/keep:2463fa9efeb75e099685528b3b9071e0+438/19.fasta.bwt'

If I change to use local inputs this works fine:

$ arvados-cwl-runner bwa-mem.cwl --reference 19.fasta.bwt --read_p1 HWI-ST1027_129_D0THKACXX.1_1.fastq --read_p2 HWI-ST1027_129_D0THKACXX.1_2.fastq --group_id arvados_tutorial --sample_id HWI-ST1027_129 --PL illumina
2017-12-28 01:34:52 cwltool INFO: /usr/bin/arvados-cwl-runner dc78526ba494973df7d298825e20503353e92adf 1.0.20171116210428, arvados-python-client 0.1.20171109204045, cwltool 1.0.20170928192020
2017-12-28 01:34:52 cwltool INFO: Resolved 'bwa-mem.cwl' to 'file:///tmp/arvados/doc/user/cwl/bwa-mem/bwa-mem.cwl'
2017-12-28 01:34:53 arvados.arv-run INFO: Upload local files: "HWI-ST1027_129_D0THKACXX.1_2.fastq" "19.fasta.amb" "19.fasta.pac" "19.fasta.ann" "HWI-ST1027_129_D0THKACXX.1_1.fastq" "19.fasta.sa" "19.fasta.bwt" 
2017-12-28 01:34:55 arvados.arv-run INFO: Uploaded to ae455ba14d14a6de72e24518781f31ba+361 (ncucu-4zz18-z1efmutce0njpt5)
2017-12-28 01:34:55 arvados.cwl-runner INFO: Uploaded to ncucu-4zz18-z71xldjxik2uko0
2017-12-28 01:34:55 arvados.cwl-runner INFO: Created pipeline ncucu-d1hrv-0xyt0iv42n1h92n
...