Arvados

ran into this with two jobs: https://cloud.curoverse.com/jobs/qr1hi-8i9sb-im16a2oic6hzlu7#Log and qr1hi-8i9sb-ybohionovr68ft

2016-08-02_19:25:36 qr1hi-8i9sb-ybohionovr68ftj 55583  running from /usr/local/arvados/src/sdk/cli/bin/crunch-job with arvados-cli Gem version(s) 0.1.20160503204200, 0.1.20151207150126, 0.1.20151023190001, 0.1.20150205181653
2016-08-02_19:25:36 qr1hi-8i9sb-ybohionovr68ftj 55583  check slurm allocation
2016-08-02_19:25:36 qr1hi-8i9sb-ybohionovr68ftj 55583  node compute0 - 16 slots
2016-08-02_19:25:36 qr1hi-8i9sb-ybohionovr68ftj 55583  start
2016-08-02_19:25:36 qr1hi-8i9sb-ybohionovr68ftj 55583  clean work dirs: start
2016-08-02_19:25:36 qr1hi-8i9sb-ybohionovr68ftj 55583  stderr starting: ['srun','--nodelist=compute0','-D','/tmp','bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk "(index(\\$3, \\"$CRUNCH_TMP\\") == 1){print \\$3}" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']
2016-08-02_19:25:37 qr1hi-8i9sb-ybohionovr68ftj 55583  clean work dirs: exit 0
2016-08-02_19:25:37 qr1hi-8i9sb-ybohionovr68ftj 55583  Install docker image c1ebfa4d67677942c223abac54a59c23+1428
2016-08-02_19:25:37 qr1hi-8i9sb-ybohionovr68ftj 55583  docker image hash is a2b612e5de55f49f5c793794db8f446165202ed6c74d7c292c21cbedf340dd02
2016-08-02_19:25:37 qr1hi-8i9sb-ybohionovr68ftj 55583  load docker image: start
2016-08-02_19:25:37 qr1hi-8i9sb-ybohionovr68ftj 55583  stderr starting: ['srun','--nodelist=compute0','/bin/bash','-o','pipefail','-ec',' if docker.io images -q --no-trunc --all | grep -xF a2b612e5de55f49f5c793794db8f446165202ed6c74d7c292c21cbedf340dd02 >/dev/null; then     exit 0 fi declare -a exit_codes=("${PIPESTATUS[@]}") if [ 0 != "${exit_codes[0]}" ]; then    exit "${exit_codes[0]}"  # `docker images` failed elif [ 1 != "${exit_codes[1]}" ]; then    exit "${exit_codes[1]}"  # `grep` encountered an error else    # Everything worked fine, but grep didn\'t find the image on this host.    arv-get c1ebfa4d67677942c223abac54a59c23\\+1428\\/a2b612e5de55f49f5c793794db8f446165202ed6c74d7c292c21cbedf340dd02\\.tar | docker.io load fi ']
2016-08-02_19:27:11 qr1hi-8i9sb-ybohionovr68ftj 55583  load docker image: exit 0
2016-08-02_19:27:11 qr1hi-8i9sb-ybohionovr68ftj 55583  check --memory-swap feature: start
2016-08-02_19:27:11 qr1hi-8i9sb-ybohionovr68ftj 55583  stderr starting: ['srun','--nodes=1','docker.io','run','--help']
2016-08-02_19:27:11 qr1hi-8i9sb-ybohionovr68ftj 55583  check --memory-swap feature: exit 0
2016-08-02_19:27:11 qr1hi-8i9sb-ybohionovr68ftj 55583  check whether default user is UID 0: start
2016-08-02_19:27:11 qr1hi-8i9sb-ybohionovr68ftj 55583  stderr starting: ['srun','--nodes=1','/bin/sh','-ec','docker.io run   a2b612e5de55f49f5c793794db8f446165202ed6c74d7c292c21cbedf340dd02 id --user']
2016-08-02_19:27:12 qr1hi-8i9sb-ybohionovr68ftj 55583  check whether default user is UID 0: exit 0
2016-08-02_19:27:12 qr1hi-8i9sb-ybohionovr68ftj 55583  check whether user 'crunch' is UID 0: start
2016-08-02_19:27:12 qr1hi-8i9sb-ybohionovr68ftj 55583  stderr starting: ['srun','--nodes=1','/bin/sh','-ec','docker.io run  --user=crunch a2b612e5de55f49f5c793794db8f446165202ed6c74d7c292c21cbedf340dd02 id --user']
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  check whether user 'crunch' is UID 0: exit 0
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  Container will run with --user=crunch
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  Looking for version 8fc5b6595fd1d544b5812d0807e9747c6eab5f8d from repository arvados
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  Using local repository '/var/lib/arvados/internal.git'
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  Version 8fc5b6595fd1d544b5812d0807e9747c6eab5f8d is commit 8fc5b6595fd1d544b5812d0807e9747c6eab5f8d
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  Failed to save git archive: git exited 128 at /usr/local/arvados/src/sdk/cli/bin/crunch-job line 2194
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  Freeze not implemented
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  collate
2016-08-02_19:27:13 qr1hi-8i9sb-ybohionovr68ftj 55583  collated output manifest text to send to API server is 0 bytes with access tokens

arvados-cwl-runner --local --verbose samtools-index-wf.cwl test-samtools-index-wf.json 
/home/bcosc/venv-test/bin/arvados-cwl-runner 1.0.20160727161315, arvados-python-client 0.1.20160727172639, cwltool 1.0.20160726135535
2016-08-02 19:22:24 arvados.cwl-runner[51648] INFO: Pipeline instance qr1hi-d1hrv-hv0d7zwf7dtzger
2016-08-02 19:22:27 arvados.cwl-runner[51648] INFO: Job samtools-index (qr1hi-8i9sb-ar0hj8kox7v8v5z) is Complete
2016-08-02 19:22:31 arvados.cwl-runner[51648] INFO: Job samtools-index_2 (qr1hi-8i9sb-3jgrylzfjcplb6y) is Complete
[step samtools-index] completion status is success
2016-08-02 19:23:19 arvados.cwl-runner[51648] INFO: Job samtools-mpileup (qr1hi-8i9sb-ybohionovr68ftj) is Queued
2016-08-02 19:23:30 arvados.cwl-runner[51648] INFO: Job samtools-mpileup_2 (qr1hi-8i9sb-im16a2oic6hzlu7) is Queued
2016-08-02 19:25:46 arvados.cwl-runner[51648] INFO: Job samtools-mpileup (qr1hi-8i9sb-ybohionovr68ftj) is Running
2016-08-02 19:25:46 arvados.cwl-runner[51648] INFO: Job samtools-mpileup_2 (qr1hi-8i9sb-im16a2oic6hzlu7) is Running
2016-08-02 19:27:16 arvados.cwl-runner[51648] INFO: Job samtools-mpileup (qr1hi-8i9sb-ybohionovr68ftj) is Failed
2016-08-02 19:27:31 arvados.cwl-runner[51648] INFO: Job samtools-mpileup_2 (qr1hi-8i9sb-im16a2oic6hzlu7) is Failed
[step samtools-mpileup] completion status is permanentFail
[workflow samtools-index-wf.cwl] outdir is $(task.outdir)
2016-08-02 19:27:32 arvados.cwl-runner[51648] WARNING: Overall process status is permanentFail
Workflow error, try again with --debug for more information:
  Workflow failed.
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/home/bcosc/venv-test/local/lib/python2.7/site-packages/arvados/events.py", line 218, in run
  File "/usr/lib/python2.7/threading.py", line 620, in wait
  File "/usr/lib/python2.7/threading.py", line 363, in wait
<type 'exceptions.ValueError'>: list.remove(x): x not in list

samtools-index-wf.cwl

cwlVersion: v1.0
class: Workflow
requirements:
  ScatterFeatureRequirement: {}
inputs:
  samtoolsindexinput:
    type: File[]
  bwa-reference:
    type: File

outputs:
  samtools-index-bam:
    type:
      type: array
      items: File
    outputSource: samtools-index/samtools-index-bam
steps:
  samtools-index:
    run: samtools-index.cwl
    scatter: samtoolsindexinput
    scatterMethod: dotproduct
    in:
      samtoolsindexinput: samtoolsindexinput
    out: [samtools-index-bam]
  samtools-mpileup:
    run: samtools-mpileup.cwl
    scatter: samtoolsmpileupinput
    scatterMethod: dotproduct
    in:
      samtoolsmpileupinput: samtools-index/samtools-index-bam
      bwa-reference: bwa-reference
    out: [samtools-mpileup-out]

test-samtools-index-wf.json

{
    "samtoolsindexinput":[
        {
          "class":"File",
          "path":"keep:9aeeae5173266894a63f7e639cb5297c+68/test_2_S8_R1.bam" 
        },
        {
          "class":"File",
          "path":"keep:5e3c920fef76581156fc1b6fe2677bca+68/test_2_S8_R1.bam" 
        }
    ],
    "bwa-reference":{
        "class":"File",
        "path":"keep:9a0815d6efd15058102a5ea01ce5c593+5838/hg19.fa" 
    },

}