2016-12-07_01:22:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  running from /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/arvados-cli-0.1.20161017193526/bin/crunch-job with arvados-cli Gem version(s) 0.1.20161017193526, 0.1.20160503204200, 0.1.20151207150126, 0.1.20151023190001
2016-12-07_01:22:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  check slurm allocation
2016-12-07_01:22:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  node humgen-05-15 - 8 slots
2016-12-07_01:22:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  start
2016-12-07_01:22:13 z8ta6-8i9sb-r52unvxj6avgrio 58297  clean work dirs: start
2016-12-07_01:22:13 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk "(index(\\$3, \\"$CRUNCH_TMP\\") == 1){print \\$3}" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']
2016-12-07_01:22:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  clean work dirs: exit 0
2016-12-07_01:22:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  Install docker image 9289c4f1fce1d1dedf9cc7876c33f936+671
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  docker image hash is 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  load docker image: start
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodelist=humgen-05-15','/bin/bash','-o','pipefail','-ec',' if /usr/bin/docker images -q --no-trunc --all | grep -xF 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 >/dev/null; then     exit 0 fi declare -a exit_codes=("${PIPESTATUS[@]}") if [ 0 != "${exit_codes[0]}" ]; then    exit "${exit_codes[0]}"  # `docker images` failed elif [ 1 != "${exit_codes[1]}" ]; then    exit "${exit_codes[1]}"  # `grep` encountered an error else    # Everything worked fine, but grep didn\'t find the image on this host.    arv-get 9289c4f1fce1d1dedf9cc7876c33f936\\+671\\/571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3\\.tar | /usr/bin/docker load fi ']
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  load docker image: exit 0
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  check --memory-swap feature: start
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodes=1','/usr/bin/docker','run','--help']
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  check --memory-swap feature: exit 0
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  check whether default user is UID 0: start
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodes=1','/bin/sh','-ec','/usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21  571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 id --user']
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  check whether default user is UID 0: exit 0
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  Container will run with default user
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  Looking for version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 from repository jr17/hgi
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  Using local repository '/var/lib/arvados/internal.git'
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  Version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 is commit 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  run install script on all workers: start
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio','sh','-c','mkdir -p /data/crunch-tmp/crunch-job/opt && cd /data/crunch-tmp/crunch-job && perl -']
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr Extracting archive: 4b8cf8e1274f8d418cb494d3ea978152
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  run install script on all workers: exit 0
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  script gatk-combinegvcfs.py
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  script_version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  script_parameters {"interval_lists_collection":"4a72f79fd2ed51ac692bc326fb168f37+10043","reference_collection":"1b447d1aa006cde57df03042736750b9+7894","inputs_collection":"b0dd82bb014b6ab8e64968ebdc07aad9+2566847"}
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  runtime_constraints {"keep_cache_mb_per_task":19500,"max_tasks_per_node":8,"min_nodes":1,"docker_image":"mercury/gatk-3.6"}
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  start level 0 with 1 slots
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 0 done, 0 running, 1 todo
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 job_task z8ta6-ot0gb-ipmynnqm3amv8n7
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 child 58563 started on humgen-05-15.1
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.0.58563','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (1 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ipmynnqm3amv8n7-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ipmynnqm3amv8n7-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ipmynnqm3amv8n7-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=0 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=0 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ipmynnqm3amv8n7 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 0 done, 1 running, 0 todo
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ipmynnqm3amv8n7-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ipmynnqm3amv8n7-0.cid --sig-proxy --memory=250853295k --memory-swap=284407723k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=0 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=0 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ipmynnqm3amv8n7 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/cgroup.procs
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/memory.stat
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4775936 rss
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/cpuacct.stat
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/cpuset.cpus
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/blkio.io_service_bytes
2016-12-07_01:22:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:22:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Processing last stream
2016-12-07_01:22:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Finalising stream .
2016-12-07_01:22:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 100_of_200
2016-12-07_01:22:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 100_of_200
2016-12-07_01:22:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 101_of_200
2016-12-07_01:22:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 101_of_200
2016-12-07_01:22:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 102_of_200
2016-12-07_01:22:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 102_of_200
2016-12-07_01:22:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 103_of_200
2016-12-07_01:22:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 103_of_200
2016-12-07_01:22:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 104_of_200
2016-12-07_01:22:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 104_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 105_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 105_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 106_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 106_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 107_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 107_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 108_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 108_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 109_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 160366592 rss
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 4.5000 user 0.4300 sys 32 cpus -- interval 10.0001 seconds 4.4700 user 0.4100 sys
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 256993 tx 3633431 rx
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 109_of_200
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 10_of_200
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 10_of_200
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 110_of_200
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 110_of_200
2016-12-07_01:22:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 111_of_200
2016-12-07_01:22:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 111_of_200
2016-12-07_01:22:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 112_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 112_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 113_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 113_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 114_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 114_of_200
2016-12-07_01:22:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 115_of_200
2016-12-07_01:22:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 115_of_200
2016-12-07_01:22:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 116_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 116_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 117_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 117_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 118_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 118_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 119_of_200
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 119_of_200
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 11_of_200
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 11_of_200
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 120_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 173228032 rss
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 6.2200 user 0.4800 sys 32 cpus -- interval 10.0000 seconds 1.7200 user 0.0500 sys
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 545008 tx 4149949 rx -- interval 10.0000 seconds 288015 tx 516518 rx
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 120_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 121_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 121_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 122_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 122_of_200
2016-12-07_01:22:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 123_of_200
2016-12-07_01:22:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 123_of_200
2016-12-07_01:22:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 124_of_200
2016-12-07_01:22:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 124_of_200
2016-12-07_01:22:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 125_of_200
2016-12-07_01:22:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 125_of_200
2016-12-07_01:22:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 126_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 126_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 127_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 127_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 128_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 128_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 129_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 129_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 12_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 12_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 130_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 130_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 131_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 131_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 132_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 132_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 133_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 133_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 134_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 134_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 135_of_200
2016-12-07_01:22:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 135_of_200
2016-12-07_01:22:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 136_of_200
2016-12-07_01:22:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 136_of_200
2016-12-07_01:22:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 137_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 137_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 138_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 138_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 139_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 139_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 13_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 13_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 140_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 140_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 141_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 179347456 rss
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 9.4900 user 0.6300 sys 32 cpus -- interval 10.0000 seconds 3.2700 user 0.1500 sys
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 1093386 tx 5073141 rx -- interval 10.0000 seconds 548378 tx 923192 rx
2016-12-07_01:22:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 141_of_200
2016-12-07_01:22:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 142_of_200
2016-12-07_01:22:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 142_of_200
2016-12-07_01:22:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 143_of_200
2016-12-07_01:22:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 143_of_200
2016-12-07_01:22:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 144_of_200
2016-12-07_01:22:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 144_of_200
2016-12-07_01:22:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 145_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 145_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 146_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 146_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 147_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 147_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 148_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 148_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 149_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 149_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 14_of_200
2016-12-07_01:22:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 14_of_200
2016-12-07_01:22:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 150_of_200
2016-12-07_01:22:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 150_of_200
2016-12-07_01:22:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 151_of_200
2016-12-07_01:22:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 151_of_200
2016-12-07_01:22:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 152_of_200
2016-12-07_01:22:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 152_of_200
2016-12-07_01:22:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 153_of_200
2016-12-07_01:22:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 153_of_200
2016-12-07_01:22:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 154_of_200
2016-12-07_01:22:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 154_of_200
2016-12-07_01:22:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 155_of_200
2016-12-07_01:22:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 155_of_200
2016-12-07_01:22:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 156_of_200
2016-12-07_01:22:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 156_of_200
2016-12-07_01:22:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 157_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 157_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 158_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 158_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 159_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 159_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 15_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 15_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 160_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 160_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 161_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 197804032 rss
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 12.4700 user 0.7600 sys 32 cpus -- interval 10.0000 seconds 2.9800 user 0.1300 sys
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 1636503 tx 5970742 rx -- interval 10.0000 seconds 543117 tx 897601 rx
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 161_of_200
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 162_of_200
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 162_of_200
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 163_of_200
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 163_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 164_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 164_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 165_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 165_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 166_of_200
2016-12-07_01:23:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 166_of_200
2016-12-07_01:23:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 167_of_200
2016-12-07_01:23:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 167_of_200
2016-12-07_01:23:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 168_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 168_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 169_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 169_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 16_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 16_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 170_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 170_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 171_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 171_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 172_of_200
2016-12-07_01:23:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 172_of_200
2016-12-07_01:23:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 173_of_200
2016-12-07_01:23:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 173_of_200
2016-12-07_01:23:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 174_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 174_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 175_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 175_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 176_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 176_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 177_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 177_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 178_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 178_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 179_of_200
2016-12-07_01:23:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 179_of_200
2016-12-07_01:23:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 17_of_200
2016-12-07_01:23:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 17_of_200
2016-12-07_01:23:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 180_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 180_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 181_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 181_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 182_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 182_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 226070528 rss
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 15.5300 user 0.9100 sys 32 cpus -- interval 10.0000 seconds 3.0600 user 0.1500 sys
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 2183539 tx 6918617 rx -- interval 10.0000 seconds 547036 tx 947875 rx
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 183_of_200
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 183_of_200
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 184_of_200
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 184_of_200
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 185_of_200
2016-12-07_01:23:10 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 185_of_200
2016-12-07_01:23:10 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 186_of_200
2016-12-07_01:23:10 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 186_of_200
2016-12-07_01:23:10 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 187_of_200
2016-12-07_01:23:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 187_of_200
2016-12-07_01:23:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 188_of_200
2016-12-07_01:23:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 188_of_200
2016-12-07_01:23:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 189_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 189_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 18_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 18_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 190_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 190_of_200
2016-12-07_01:23:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 191_of_200
2016-12-07_01:23:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 191_of_200
2016-12-07_01:23:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 192_of_200
2016-12-07_01:23:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 192_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 193_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 193_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 194_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 194_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 195_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 195_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 196_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 196_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 197_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 197_of_200
2016-12-07_01:23:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 198_of_200
2016-12-07_01:23:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 198_of_200
2016-12-07_01:23:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 199_of_200
2016-12-07_01:23:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 199_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 19_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 19_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 1_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 1_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 200_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 200_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 20_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 20_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 21_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 254140416 rss
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 18.5000 user 1.0700 sys 32 cpus -- interval 10.0000 seconds 2.9700 user 0.1600 sys
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 2707575 tx 7815380 rx -- interval 10.0000 seconds 524036 tx 896763 rx
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 21_of_200
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 22_of_200
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 22_of_200
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 23_of_200
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 23_of_200
2016-12-07_01:23:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 24_of_200
2016-12-07_01:23:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 24_of_200
2016-12-07_01:23:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 25_of_200
2016-12-07_01:23:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 25_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 26_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 26_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 27_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 27_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 28_of_200
2016-12-07_01:23:22 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 28_of_200
2016-12-07_01:23:22 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 29_of_200
2016-12-07_01:23:22 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 29_of_200
2016-12-07_01:23:22 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 2_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 2_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 30_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 30_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 31_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 31_of_200
2016-12-07_01:23:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 32_of_200
2016-12-07_01:23:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 32_of_200
2016-12-07_01:23:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 33_of_200
2016-12-07_01:23:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 33_of_200
2016-12-07_01:23:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 34_of_200
2016-12-07_01:23:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 34_of_200
2016-12-07_01:23:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 35_of_200
2016-12-07_01:23:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 35_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 36_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 36_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 37_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 37_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 38_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 38_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 39_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 39_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 3_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 3_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 40_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 40_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 41_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 41_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 280858624 rss
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 21.4900 user 1.2400 sys 32 cpus -- interval 10.0000 seconds 2.9900 user 0.1700 sys
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 3250753 tx 8733455 rx -- interval 10.0000 seconds 543178 tx 918075 rx
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 42_of_200
2016-12-07_01:23:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 42_of_200
2016-12-07_01:23:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 43_of_200
2016-12-07_01:23:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 43_of_200
2016-12-07_01:23:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 44_of_200
2016-12-07_01:23:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 44_of_200
2016-12-07_01:23:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 45_of_200
2016-12-07_01:23:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 45_of_200
2016-12-07_01:23:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 46_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 46_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 47_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 47_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 48_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 48_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 49_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 49_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 4_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 4_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 50_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 50_of_200
2016-12-07_01:23:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 51_of_200
2016-12-07_01:23:36 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 51_of_200
2016-12-07_01:23:36 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 52_of_200
2016-12-07_01:23:36 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 52_of_200
2016-12-07_01:23:36 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 53_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 53_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 54_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 54_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 55_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 55_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 56_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 56_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 57_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 57_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 58_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 302202880 rss
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 23.8300 user 1.3400 sys 32 cpus -- interval 10.0000 seconds 2.3400 user 0.1000 sys
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 3675202 tx 9431114 rx -- interval 10.0000 seconds 424449 tx 697659 rx
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 58_of_200
2016-12-07_01:23:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 59_of_200
2016-12-07_01:23:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 59_of_200
2016-12-07_01:23:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 5_of_200
2016-12-07_01:23:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 5_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 60_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 60_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 61_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 61_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 62_of_200
2016-12-07_01:23:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 62_of_200
2016-12-07_01:23:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 63_of_200
2016-12-07_01:23:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 63_of_200
2016-12-07_01:23:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 64_of_200
2016-12-07_01:23:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 64_of_200
2016-12-07_01:23:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 65_of_200
2016-12-07_01:23:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 65_of_200
2016-12-07_01:23:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 66_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 66_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 67_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 67_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 68_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 68_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 69_of_200
2016-12-07_01:23:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 69_of_200
2016-12-07_01:23:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 6_of_200
2016-12-07_01:23:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 6_of_200
2016-12-07_01:23:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 70_of_200
2016-12-07_01:23:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 70_of_200
2016-12-07_01:23:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 71_of_200
2016-12-07_01:23:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 71_of_200
2016-12-07_01:23:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 72_of_200
2016-12-07_01:23:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 72_of_200
2016-12-07_01:23:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 73_of_200
2016-12-07_01:23:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 73_of_200
2016-12-07_01:23:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 74_of_200
2016-12-07_01:23:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 74_of_200
2016-12-07_01:23:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 75_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 75_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 76_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 76_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 77_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 77_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 309436416 rss
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 27.1400 user 1.6200 sys 32 cpus -- interval 10.0000 seconds 3.3100 user 0.2800 sys
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 4190913 tx 10309781 rx -- interval 10.0000 seconds 515711 tx 878667 rx
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 78_of_200
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 78_of_200
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 79_of_200
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 79_of_200
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 7_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 7_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 80_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 80_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 81_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 81_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 82_of_200
2016-12-07_01:23:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 82_of_200
2016-12-07_01:23:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 83_of_200
2016-12-07_01:23:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 83_of_200
2016-12-07_01:23:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 84_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 84_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 85_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 85_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 86_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 86_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 87_of_200
2016-12-07_01:23:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 87_of_200
2016-12-07_01:23:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 88_of_200
2016-12-07_01:23:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 88_of_200
2016-12-07_01:23:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 89_of_200
2016-12-07_01:23:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 89_of_200
2016-12-07_01:23:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 8_of_200
2016-12-07_01:23:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 8_of_200
2016-12-07_01:23:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 90_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 90_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 91_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 91_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 92_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 92_of_200
2016-12-07_01:23:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 93_of_200
2016-12-07_01:23:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 93_of_200
2016-12-07_01:23:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 94_of_200
2016-12-07_01:23:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 94_of_200
2016-12-07_01:23:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 95_of_200
2016-12-07_01:23:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 95_of_200
2016-12-07_01:23:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 96_of_200
2016-12-07_01:23:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 96_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 97_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 97_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 98_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 98_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 309436416 rss
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 30.2200 user 1.7400 sys 32 cpus -- interval 10.0000 seconds 3.0800 user 0.1200 sys
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 4735428 tx 11234763 rx -- interval 10.0000 seconds 544515 tx 924982 rx
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 99_of_200
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 99_of_200
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 9_of_200
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 9_of_200
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Ending task 0 successfully
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 child 58563 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 success in 102 seconds
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 task output (0 bytes): 
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 200 new tasks from API server.
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 1 done, 0 running, 200 todo
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  wait for last 0 children to finish
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  start level 1 with 8 slots
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 1 done, 0 running, 200 todo
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 job_task z8ta6-ot0gb-n1ij7gzepmmyrnw
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 child 61926 started on humgen-05-15.1
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 job_task z8ta6-ot0gb-3xe7jgu01d1yiud
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 child 61939 started on humgen-05-15.2
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 job_task z8ta6-ot0gb-q5vot4nnen1ql8m
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 child 61943 started on humgen-05-15.3
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 job_task z8ta6-ot0gb-wc4e7gbasnzlk4n
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 child 61954 started on humgen-05-15.4
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 job_task z8ta6-ot0gb-380otvu8nf013pi
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 child 61959 started on humgen-05-15.5
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 job_task z8ta6-ot0gb-8vdh6ynyoup41n6
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 child 61963 started on humgen-05-15.6
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 job_task z8ta6-ot0gb-i83578e97ax5esa
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 child 61975 started on humgen-05-15.7
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 job_task z8ta6-ot0gb-ab4wzw3us93bwkz
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 child 61979 started on humgen-05-15.8
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.1.61926','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-n1ij7gzepmmyrnw-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-n1ij7gzepmmyrnw-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-n1ij7gzepmmyrnw-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=1 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-n1ij7gzepmmyrnw --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-n1ij7gzepmmyrnw-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-n1ij7gzepmmyrnw-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=1 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-n1ij7gzepmmyrnw --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/cgroup.procs
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/memory.stat
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5513216 rss
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/cpuacct.stat
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/cpuset.cpus
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/blkio.io_service_bytes
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.2.61939','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-3xe7jgu01d1yiud-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-3xe7jgu01d1yiud-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-3xe7jgu01d1yiud-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=2 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-3xe7jgu01d1yiud --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.3.61943','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5vot4nnen1ql8m-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-q5vot4nnen1ql8m-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5vot4nnen1ql8m-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=3 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-q5vot4nnen1ql8m --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.4.61954','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-wc4e7gbasnzlk4n-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-wc4e7gbasnzlk4n-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-wc4e7gbasnzlk4n-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=4 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-wc4e7gbasnzlk4n --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.5.61959','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-380otvu8nf013pi-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-380otvu8nf013pi-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-380otvu8nf013pi-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=5 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-380otvu8nf013pi --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.6.61963','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8vdh6ynyoup41n6-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8vdh6ynyoup41n6-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8vdh6ynyoup41n6-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=6 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-8vdh6ynyoup41n6 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.7.61975','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i83578e97ax5esa-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-i83578e97ax5esa-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i83578e97ax5esa-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=7 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-i83578e97ax5esa --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.8.61979','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ab4wzw3us93bwkz-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ab4wzw3us93bwkz-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ab4wzw3us93bwkz-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=8 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ab4wzw3us93bwkz --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 1 done, 8 running, 192 todo
2016-12-07_01:24:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:24:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Mounting task input collection to get interval_list
2016-12-07_01:24:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Interval_List collection mounted at /keep/812316f6c95d8641412912523842bbfc+94
2016-12-07_01:24:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Total chunk length is 15618872
2016-12-07_01:24:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:24:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Have 1 intervals
2016-12-07_01:24:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:24:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:24:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:24:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:24:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Have 5400 potential reusable task outputs
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Have 5400 potentially reusable tasks
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Creating task to process interval: [9:6148617-21767488]
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr No reusable JobTask matched key parameters [u'100_of_200', u'a241297995deeab501f9bd99dc533315+14700', '9:6148617-21767488', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Ending task 1 successfully
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 child 61926 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 success in 9 seconds
2016-12-07_01:24:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 task output (0 bytes): 
2016-12-07_01:24:21 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:24:27 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 2 done, 7 running, 193 todo
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 job_task z8ta6-ot0gb-eeqomjqy20rc5ff
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 child 62498 started on humgen-05-15.1
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.9.62498','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-eeqomjqy20rc5ff-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-eeqomjqy20rc5ff-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-eeqomjqy20rc5ff-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=9 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-eeqomjqy20rc5ff --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-3xe7jgu01d1yiud-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-3xe7jgu01d1yiud-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=2 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-3xe7jgu01d1yiud --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/cgroup.procs
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/memory.stat
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: mem 69632 cache 0 swap 0 pgmajfault 6004736 rss
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/cpuacct.stat
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/cpuset.cpus
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/blkio.io_service_bytes
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Mounting task input collection to get interval_list
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Interval_List collection mounted at /keep/6af141a235ccb024f6a32fc12ee98dcf+94
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Total chunk length is 15618872
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Have 1 intervals
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 2 done, 8 running, 192 todo
2016-12-07_01:24:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Have 5400 potential reusable task outputs
2016-12-07_01:24:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Have 5400 potentially reusable tasks
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Creating task to process interval: [9:21767489-37386360]
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr No reusable JobTask matched key parameters [u'101_of_200', u'1b848222e216cbae87e6932d5820a285+14700', '9:21767489-37386360', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Ending task 1 successfully
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid 52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43, parent docker, root /sys/fs/cgroup
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid 52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43, parent docker, root /sys/fs/cgroup
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid 52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43, parent docker, root /sys/fs/cgroup
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid 52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43, parent docker, root /sys/fs/cgroup
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr srun: Job step created
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 child 61939 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 success in 20 seconds
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 task output (0 bytes): 
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 3 done, 7 running, 193 todo
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 job_task z8ta6-ot0gb-73kg9lkqq5037a4
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 child 62816 started on humgen-05-15.2
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.10.62816','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-73kg9lkqq5037a4-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-73kg9lkqq5037a4-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-73kg9lkqq5037a4-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=10 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-73kg9lkqq5037a4 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-q5vot4nnen1ql8m-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5vot4nnen1ql8m-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=3 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-q5vot4nnen1ql8m --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/cgroup.procs
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/memory.stat
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 3395584 rss
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/cpuacct.stat
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/cpuset.cpus
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: cpu 0.0200 user 0.0100 sys 32 cpus
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/blkio.io_service_bytes
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Mounting task input collection to get interval_list
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Interval_List collection mounted at /keep/2e7cf8719961c4cc5b2ffdd92ff99dd0+94
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Total chunk length is 15618872
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Have 1 intervals
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 3 done, 8 running, 192 todo
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: cpu 0.5200 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.5000 user 0.1500 sys
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: net:eth0 21985 tx 238258 rx
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Have 5400 potential reusable task outputs
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Have 5400 potentially reusable tasks
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Creating task to process interval: [9:37386361-53005232]
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr No reusable JobTask matched key parameters [u'102_of_200', u'0b01f4aa638023c7844e14aa840fe4cc+14532', '9:37386361-53005232', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Ending task 1 successfully
2016-12-07_01:24:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 child 61943 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:24:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 success in 33 seconds
2016-12-07_01:24:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 task output (0 bytes): 
2016-12-07_01:24:44 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 4 done, 7 running, 193 todo
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 job_task z8ta6-ot0gb-q5ob5ca6mm8v0v8
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 child 63162 started on humgen-05-15.3
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.11.63162','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5ob5ca6mm8v0v8-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-q5ob5ca6mm8v0v8-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5ob5ca6mm8v0v8-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=11 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-q5ob5ca6mm8v0v8 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr srun: Job step created
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-380otvu8nf013pi-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-380otvu8nf013pi-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=5 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-380otvu8nf013pi --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/cgroup.procs
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/memory.stat
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4403200 rss
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/cpuacct.stat
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/cpuset.cpus
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/blkio.io_service_bytes
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Mounting task input collection to get interval_list
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Interval_List collection mounted at /keep/61144aa3079bc748eb474251bb12d2af+94
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Total chunk length is 15618872
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Have 1 intervals
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 4 done, 8 running, 192 todo
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Have 5400 potential reusable task outputs
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Have 5400 potentially reusable tasks
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Creating task to process interval: [9:68624105-84242976]
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr No reusable JobTask matched key parameters [u'104_of_200', u'0e7a8a6d12ed4d9a7f7a47c972f98c08+14700', '9:68624105-84242976', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Ending task 1 successfully
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 child 61959 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 success in 43 seconds
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 task output (0 bytes): 
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 5 done, 7 running, 193 todo
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 job_task z8ta6-ot0gb-kbnkrhgu14765w0
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 child 63553 started on humgen-05-15.5
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr srun: Job step created
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-eeqomjqy20rc5ff-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-eeqomjqy20rc5ff-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=9 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-eeqomjqy20rc5ff --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/cgroup.procs
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/memory.stat
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7778304 rss
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/cpuacct.stat
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/cpuset.cpus
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: cpu 0.0100 user 0.0000 sys 32 cpus
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/blkio.io_service_bytes
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Interval_List collection mounted at /keep/64696f9b403b7674c728386882cbd3b7+94
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Total chunk length is 15498872
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Have 1 intervals
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.12.63553','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kbnkrhgu14765w0-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-kbnkrhgu14765w0-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kbnkrhgu14765w0-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=12 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-kbnkrhgu14765w0 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 5 done, 8 running, 192 todo
2016-12-07_01:25:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: blkio:0:0 0 write 12564 read -- interval 10.0000 seconds 0 write 12564 read
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Have 5400 potential reusable task outputs
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46039040 rss
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: cpu 0.6600 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.6500 user 0.1800 sys
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: net:eth0 34851 tx 3237537 rx
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Have 5400 potentially reusable tasks
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Creating task to process interval: [9:131099593-141213431 10:1-5385033]
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr No reusable JobTask matched key parameters [u'108_of_200', u'7e64dee294f55ab4bfb56c1a154618f8+14704', '9:131099593-141213431 10:1-5385033', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Ending task 1 successfully
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 child 62498 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 success in 37 seconds
2016-12-07_01:25:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 task output (0 bytes): 
2016-12-07_01:25:06 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 6 done, 7 running, 193 todo
2016-12-07_01:25:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 job_task z8ta6-ot0gb-0r2ariip11y1uwg
2016-12-07_01:25:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 child 64183 started on humgen-05-15.1
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.13.64183','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0r2ariip11y1uwg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0r2ariip11y1uwg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0r2ariip11y1uwg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=13 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-0r2ariip11y1uwg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr srun: Job step created
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8vdh6ynyoup41n6-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8vdh6ynyoup41n6-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=6 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-8vdh6ynyoup41n6 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/cgroup.procs
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/memory.stat
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5545984 rss
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/cpuacct.stat
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/cpuset.cpus
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/blkio.io_service_bytes
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Interval_List collection mounted at /keep/4325c7dac5c39f2db578e849695a12aa+94
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Total chunk length is 15618872
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Have 1 intervals
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 6 done, 8 running, 192 todo
2016-12-07_01:25:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Have 5400 potential reusable task outputs
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 notice: task is not in slurm queue but srun process 61979 has not exited
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 notice: task is not in slurm queue but srun process 61975 has not exited
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 notice: task is not in slurm queue but srun process 61954 has not exited
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Have 5400 potentially reusable tasks
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Creating task to process interval: [9:84242977-99861848]
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr No reusable JobTask matched key parameters [u'105_of_200', u'3a3bb24e84ba30a06b3d46caa3b1c040+14700', '9:84242977-99861848', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Ending task 1 successfully
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid 7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid 7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid 7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid 7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr srun: Job step created
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 child 61963 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 success in 65 seconds
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 task output (0 bytes): 
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 7 done, 7 running, 193 todo
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 job_task z8ta6-ot0gb-b9qjnxrsbhkkm8l
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 child 64510 started on humgen-05-15.6
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-q5ob5ca6mm8v0v8-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5ob5ca6mm8v0v8-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=11 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-q5ob5ca6mm8v0v8 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/cgroup.procs
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/memory.stat
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7946240 rss
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/cpuacct.stat
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/cpuset.cpus
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/blkio.io_service_bytes
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Interval_List collection mounted at /keep/47924fc036d6a931b5f0cf53440e0969+92
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Total chunk length is 15618872
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Have 1 intervals
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.14.64510','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-b9qjnxrsbhkkm8l-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-b9qjnxrsbhkkm8l-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-b9qjnxrsbhkkm8l-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=14 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-b9qjnxrsbhkkm8l --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:24 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 7 done, 8 running, 192 todo
2016-12-07_01:25:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Have 5400 potential reusable task outputs
2016-12-07_01:25:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Have 5400 potentially reusable tasks
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Creating task to process interval: [1:140569849-156188720]
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr No reusable JobTask matched key parameters [u'10_of_200', u'a99a25ff7de535672c0817451b92bd1a+14616', '1:140569849-156188720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Ending task 1 successfully
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 child 63162 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 success in 37 seconds
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 task output (0 bytes): 
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 8 done, 7 running, 193 todo
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 job_task z8ta6-ot0gb-k3exaksrtpevoki
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 child 64881 started on humgen-05-15.3
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.15.64881','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k3exaksrtpevoki-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-k3exaksrtpevoki-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k3exaksrtpevoki-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=15 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-k3exaksrtpevoki --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr srun: Job step created
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-i83578e97ax5esa-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i83578e97ax5esa-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=7 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-i83578e97ax5esa --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/cgroup.procs
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/memory.stat
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: mem 16384 cache 0 swap 0 pgmajfault 5697536 rss
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/cpuacct.stat
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/cpuset.cpus
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/blkio.io_service_bytes
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: net:eth0 0 tx 0 rx
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Interval_List collection mounted at /keep/34f0119dab554faf0acfc04f3cae7c1d+94
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Total chunk length is 15618872
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Have 1 intervals
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:35 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 8 done, 8 running, 192 todo
2016-12-07_01:25:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24121344 rss
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: cpu 0.5000 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.4600 user 0.1600 sys
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: net:eth0 22009 tx 238347 rx -- interval 10.0002 seconds 22009 tx 238347 rx
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Have 5400 potential reusable task outputs
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Have 5400 potentially reusable tasks
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Creating task to process interval: [9:99861849-115480720]
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr No reusable JobTask matched key parameters [u'106_of_200', u'dc8fa207ac10a02e00e1ba8ab1687ad9+14700', '9:99861849-115480720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Ending task 1 successfully
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr srun: Job step created
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 child 61975 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 success in 88 seconds
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 task output (0 bytes): 
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 9 done, 7 running, 193 todo
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 job_task z8ta6-ot0gb-204sempjbouqi2x
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 child 65269 started on humgen-05-15.7
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0r2ariip11y1uwg-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0r2ariip11y1uwg-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=13 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-0r2ariip11y1uwg --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/cgroup.procs
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/memory.stat
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4087808 rss
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/cpuacct.stat
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/cpuset.cpus
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/blkio.io_service_bytes
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Interval_List collection mounted at /keep/d6cf1afdecfe2ae2261292a571a3759b+94
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Total chunk length is 15618872
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Have 1 intervals
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.16.65269','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-204sempjbouqi2x-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-204sempjbouqi2x-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-204sempjbouqi2x-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=16 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-204sempjbouqi2x --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 9 done, 8 running, 192 todo
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 killing orphaned srun process 61979 (task not in slurm queue, no stderr received in last 89s)
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 sending 2x signal 2 to pid 61979
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 killing orphaned srun process 61954 (task not in slurm queue, no stderr received in last 89s)
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 sending 2x signal 2 to pid 61954
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 notice: task is not in slurm queue but srun process 62816 has not exited
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 child 61954 on humgen-05-15.4 exit 1 success=
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:25:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 failure (#1, temporary) after 98 seconds
2016-12-07_01:25:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 task output (0 bytes): 
2016-12-07_01:25:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 child 61979 on humgen-05-15.8 exit 1 success=
2016-12-07_01:25:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 failure (#1, temporary) after 100 seconds
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 task output (0 bytes): 
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 child 64183 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 success in 41 seconds
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 task output (0 bytes): 
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297  3 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:59 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:59 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 10 done, 5 running, 195 todo
2016-12-07_01:25:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 job_task z8ta6-ot0gb-ttjug7tm91r0b32
2016-12-07_01:25:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 child 465 started on humgen-05-15.4
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 job_task z8ta6-ot0gb-1fbv3vnkrx4i2ix
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 child 473 started on humgen-05-15.8
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 job_task z8ta6-ot0gb-5n5f89qjphfwamo
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 child 477 started on humgen-05-15.1
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.17.465','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ttjug7tm91r0b32-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ttjug7tm91r0b32-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ttjug7tm91r0b32-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=17 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ttjug7tm91r0b32 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.18.473','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1fbv3vnkrx4i2ix-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-1fbv3vnkrx4i2ix-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1fbv3vnkrx4i2ix-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=18 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-1fbv3vnkrx4i2ix --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr srun: Job step created
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-kbnkrhgu14765w0-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kbnkrhgu14765w0-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=12 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-kbnkrhgu14765w0 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/cgroup.procs
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/memory.stat
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7962624 rss
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/cpuacct.stat
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/cpuset.cpus
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: cpu 0.0200 user 0.0100 sys 32 cpus
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/blkio.io_service_bytes
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Interval_List collection mounted at /keep/11837032d32bcfff7055a484f9c0ddb8+94
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Total chunk length is 15618872
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Have 1 intervals
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.19.477','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5n5f89qjphfwamo-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-5n5f89qjphfwamo-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5n5f89qjphfwamo-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=19 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-5n5f89qjphfwamo --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 10 done, 8 running, 192 todo
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Have 5400 potentially reusable tasks
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Creating task to process interval: [10:21003906-36622777]
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr No reusable JobTask matched key parameters [u'110_of_200', u'e20f5f00b25cd7e53f799770dbe9e308+14700', '10:21003906-36622777', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Ending task 1 successfully
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr srun: Job step created
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 child 63553 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 success in 61 seconds
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 task output (0 bytes): 
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 11 done, 7 running, 193 todo
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 job_task z8ta6-ot0gb-xiz6ocqs0kyyqzj
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 child 796 started on humgen-05-15.5
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.20.796','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xiz6ocqs0kyyqzj-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xiz6ocqs0kyyqzj-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xiz6ocqs0kyyqzj-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=20 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-xiz6ocqs0kyyqzj --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-204sempjbouqi2x-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-204sempjbouqi2x-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=16 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-204sempjbouqi2x --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/cgroup.procs
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/memory.stat
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: mem 94208 cache 0 swap 0 pgmajfault 5992448 rss
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/cpuacct.stat
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/cpuset.cpus
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/blkio.io_service_bytes
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Interval_List collection mounted at /keep/8b0a6de6be0a7b3dd7360336477e02fd+94
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Total chunk length is 15618872
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Have 1 intervals
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 11 done, 8 running, 192 todo
2016-12-07_01:26:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Have 5400 potentially reusable tasks
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Creating task to process interval: [10:83479394-99098265]
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr No reusable JobTask matched key parameters [u'114_of_200', u'c3bc3b07edcf5d2b7b9bfb2b5154d56f+14700', '10:83479394-99098265', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Ending task 1 successfully
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 child 65269 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 success in 26 seconds
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 task output (0 bytes): 
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 12 done, 7 running, 193 todo
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 job_task z8ta6-ot0gb-2vh6dr8bmvpcz1x
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 child 1139 started on humgen-05-15.7
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ttjug7tm91r0b32-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ttjug7tm91r0b32-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=17 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ttjug7tm91r0b32 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/cgroup.procs
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/memory.stat
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5664768 rss
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/cpuacct.stat
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/cpuset.cpus
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/blkio.io_service_bytes
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Interval_List collection mounted at /keep/e05c9a56f87bbb0e87bd78edfb2138bb+94
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Total chunk length is 15618872
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Have 1 intervals
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.21.1139','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2vh6dr8bmvpcz1x-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2vh6dr8bmvpcz1x-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2vh6dr8bmvpcz1x-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=21 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2vh6dr8bmvpcz1x --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 12 done, 8 running, 192 todo
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: blkio:0:0 0 write 12527 read -- interval 10.0000 seconds 0 write 12527 read
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24084480 rss
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: cpu 0.5000 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4600 user 0.1500 sys
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: net:eth0 22273 tx 238495 rx
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 killing orphaned srun process 62816 (task not in slurm queue, no stderr received in last 94s)
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 sending 2x signal 2 to pid 62816
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 notice: task is not in slurm queue but srun process 64510 has not exited
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 child 62816 on humgen-05-15.2 exit 1 success=
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 failure (#1, temporary) after 106 seconds
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 task output (0 bytes): 
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 child 465 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 success in 27 seconds
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 task output (0 bytes): 
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297  2 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:32 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 13 done, 6 running, 194 todo
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 job_task z8ta6-ot0gb-8zpggr6kb52w373
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 child 1517 started on humgen-05-15.2
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 job_task z8ta6-ot0gb-1uoyxunyt78l0t9
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 child 1522 started on humgen-05-15.4
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.22.1517','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8zpggr6kb52w373-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8zpggr6kb52w373-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8zpggr6kb52w373-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=22 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-8zpggr6kb52w373 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.23.1522','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1uoyxunyt78l0t9-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-1uoyxunyt78l0t9-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1uoyxunyt78l0t9-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=23 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-1uoyxunyt78l0t9 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xiz6ocqs0kyyqzj-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xiz6ocqs0kyyqzj-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=20 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-xiz6ocqs0kyyqzj --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/cgroup.procs
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/memory.stat
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: mem 94208 cache 0 swap 0 pgmajfault 6393856 rss
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/cpuacct.stat
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/cpuset.cpus
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/blkio.io_service_bytes
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Interval_List collection mounted at /keep/d9aa702cd7b3a1e6b6dec54b4a3d8e35+94
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Total chunk length is 15618872
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Have 1 intervals
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 13 done, 8 running, 192 todo
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Have 5400 potentially reusable tasks
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Creating task to process interval: [11:10300135-25919006]
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr No reusable JobTask matched key parameters [u'118_of_200', u'79b22e2a5c2a107fd090c78ed23ff2e7+14700', '11:10300135-25919006', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Ending task 1 successfully
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr srun: Job step created
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 child 796 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 success in 26 seconds
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 task output (0 bytes): 
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 14 done, 7 running, 193 todo
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 job_task z8ta6-ot0gb-2r646fu5fbu5irj
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 child 1854 started on humgen-05-15.5
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-k3exaksrtpevoki-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k3exaksrtpevoki-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=15 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-k3exaksrtpevoki --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/cgroup.procs
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/memory.stat
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 884736 rss
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/cpuacct.stat
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/cpuset.cpus
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: cpu 0.0100 user 0.0000 sys 32 cpus
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/blkio.io_service_bytes
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Interval_List collection mounted at /keep/f73eab8bfca061b956c4cbaa59d62ccd+94
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Total chunk length is 15618872
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Have 1 intervals
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.24.1854','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2r646fu5fbu5irj-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2r646fu5fbu5irj-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2r646fu5fbu5irj-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=24 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2r646fu5fbu5irj --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 14 done, 8 running, 192 todo
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Have 5400 potentially reusable tasks
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Creating task to process interval: [10:67860522-83479393]
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr No reusable JobTask matched key parameters [u'113_of_200', u'45d9d4c732c4ecae90a7f5ec11993512+14700', '10:67860522-83479393', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Ending task 1 successfully
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:26:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 child 64881 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:26:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 success in 72 seconds
2016-12-07_01:26:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 task output (0 bytes): 
2016-12-07_01:26:46 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:52 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 15 done, 7 running, 193 todo
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 job_task z8ta6-ot0gb-h0sz2rle16hsfng
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 child 2495 started on humgen-05-15.3
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-1uoyxunyt78l0t9-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1uoyxunyt78l0t9-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=23 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-1uoyxunyt78l0t9 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/cgroup.procs
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/memory.stat
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5427200 rss
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/cpuacct.stat
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/cpuset.cpus
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/blkio.io_service_bytes
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Interval_List collection mounted at /keep/b34c79e916b3f4868c677e831b11c26d+94
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Total chunk length is 15618872
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Have 1 intervals
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.25.2495','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h0sz2rle16hsfng-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-h0sz2rle16hsfng-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h0sz2rle16hsfng-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=25 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-h0sz2rle16hsfng --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 15 done, 8 running, 192 todo
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: cpu 0.4700 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.4400 user 0.1700 sys
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: net:eth0 21853 tx 238207 rx
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 killing orphaned srun process 64510 (task not in slurm queue, no stderr received in last 83s)
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 sending 2x signal 2 to pid 64510
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 child 64510 on humgen-05-15.6 exit 1 success=
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 failure (#1, temporary) after 94 seconds
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 task output (0 bytes): 
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 child 1522 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 success in 26 seconds
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 task output (0 bytes): 
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297  2 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 16 done, 6 running, 194 todo
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297  Every node has failed -- giving up
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297  wait for last 6 children to finish
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr srun: Job step created
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-1fbv3vnkrx4i2ix-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1fbv3vnkrx4i2ix-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=18 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-1fbv3vnkrx4i2ix --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/cgroup.procs
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/memory.stat
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4861952 rss
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/cpuacct.stat
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/cpuset.cpus
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/blkio.io_service_bytes
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Interval_List collection mounted at /keep/359a96b521b56e8d8e36b57450fa80ac+94
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Total chunk length is 15618872
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Have 1 intervals
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:27:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Creating task to process interval: [10:114717138-130336009]
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr No reusable JobTask matched key parameters [u'116_of_200', u'6365525c0ee094fee362794dbbab6a30+14700', '10:114717138-130336009', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Ending task 1 successfully
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: blkio:0:0 0 write 12528 read -- interval 10.0000 seconds 0 write 12528 read
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 child 473 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 success in 68 seconds
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 task output (0 bytes): 
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-h0sz2rle16hsfng-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h0sz2rle16hsfng-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=25 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-h0sz2rle16hsfng --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/cgroup.procs
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/memory.stat
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 1810432 rss
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/cpuacct.stat
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/cpuset.cpus
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: cpu 0.0200 user 0.0200 sys 32 cpus
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/blkio.io_service_bytes
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Interval_List collection mounted at /keep/bbfdffd38f1432cef552ced110ab3a90+94
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Total chunk length is 15618872
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Have 1 intervals
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 notice: task is not in slurm queue but srun process 477 has not exited
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 17 done, 5 running, 195 todo
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:27:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:27:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: cpu 0.5000 user 0.1800 sys 32 cpus -- interval 10.0001 seconds 0.4800 user 0.1600 sys
2016-12-07_01:27:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: net:eth0 21919 tx 238193 rx
2016-12-07_01:27:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Creating task to process interval: [11:72775623-88394494]
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr No reusable JobTask matched key parameters [u'122_of_200', u'af62e8edc576e4baf02413c2134c9270+14700', '11:72775623-88394494', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Ending task 1 successfully
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 child 2495 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 success in 27 seconds
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 task output (0 bytes): 
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr srun: Job step created
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-5n5f89qjphfwamo-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5n5f89qjphfwamo-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=19 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-5n5f89qjphfwamo --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/cgroup.procs
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/memory.stat
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 258048 rss
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/cpuacct.stat
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/cpuset.cpus
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: cpu 0.0100 user 0.0100 sys 32 cpus
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/blkio.io_service_bytes
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Interval_List collection mounted at /keep/76a230c91cc5d0053b75306d58906b2a+94
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Total chunk length is 15498872
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Have 1 intervals
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 18 done, 4 running, 196 todo
2016-12-07_01:27:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 notice: task is not in slurm queue but srun process 1139 has not exited
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: blkio:0:0 0 write 12567 read -- interval 10.0000 seconds 0 write 12567 read
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24084480 rss
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: cpu 0.5200 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.5100 user 0.1400 sys
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: net:eth0 21807 tx 238026 rx
2016-12-07_01:27:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Creating task to process interval: [10:130336010-135534747 11:1-10300134]
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr No reusable JobTask matched key parameters [u'117_of_200', u'a986cd76d508917b0b96bd60c6107961+14704', '10:130336010-135534747 11:1-10300134', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Ending task 1 successfully
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr srun: Job step created
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 child 477 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 success in 93 seconds
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 task output (0 bytes): 
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2r646fu5fbu5irj-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2r646fu5fbu5irj-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=24 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-2r646fu5fbu5irj --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/cgroup.procs
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/memory.stat
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4861952 rss
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/cpuacct.stat
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/cpuset.cpus
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/blkio.io_service_bytes
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Interval_List collection mounted at /keep/dcd0caa42cf81cfae2c2a7a7d6a5503a+94
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Total chunk length is 15618872
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Have 1 intervals
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 19 done, 3 running, 197 todo
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: cpu 0.5200 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.4800 user 0.1400 sys
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: net:eth0 22051 tx 238193 rx
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Creating task to process interval: [11:57156751-72775622]
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr No reusable JobTask matched key parameters [u'121_of_200', u'7770dae0a3517e6dca33c49e54e80074+14704', '11:57156751-72775622', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Ending task 1 successfully
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 notice: task is not in slurm queue but srun process 1517 has not exited
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr srun: Job step created
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 child 1854 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 success in 63 seconds
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 task output (0 bytes): 
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2vh6dr8bmvpcz1x-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2vh6dr8bmvpcz1x-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=21 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-2vh6dr8bmvpcz1x --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/cgroup.procs
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/memory.stat
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5664768 rss
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/cpuacct.stat
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/cpuset.cpus
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/blkio.io_service_bytes
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Interval_List collection mounted at /keep/dbba1077cec452fe00018827674a6082+94
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Total chunk length is 15618872
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Have 1 intervals
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 20 done, 2 running, 198 todo
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Creating task to process interval: [11:25919007-41537878]
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr No reusable JobTask matched key parameters [u'119_of_200', u'e715710b28adf5479c2b4ee6a270292a+14700', '11:25919007-41537878', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Ending task 1 successfully
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 52002816 rss
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: cpu 0.7800 user 0.1900 sys 32 cpus -- interval 10.0002 seconds 0.7400 user 0.1900 sys
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: net:eth0 40222 tx 3250421 rx
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 child 1139 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 success in 97 seconds
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 task output (0 bytes): 
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr srun: Job step created
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8zpggr6kb52w373-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8zpggr6kb52w373-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=22 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-8zpggr6kb52w373 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/cgroup.procs
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/memory.stat
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5410816 rss
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/cpuacct.stat
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/cpuset.cpus
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/blkio.io_service_bytes
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Mounting task input collection to get interval_list
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Interval_List collection mounted at /keep/a19791935337560ca00961b9d0c50277+92
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Total chunk length is 15618872
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Have 1 intervals
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 21 done, 1 running, 199 todo
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Have 5400 potential reusable task outputs
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46022656 rss
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: cpu 0.6800 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.6400 user 0.1800 sys
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: net:eth0 35181 tx 3235991 rx
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Have 5400 potentially reusable tasks
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Creating task to process interval: [1:156188721-171807592]
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr No reusable JobTask matched key parameters [u'11_of_200', u'0407d47186859c17195bd2efb49797bb+14616', '1:156188721-171807592', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Ending task 1 successfully
2016-12-07_01:28:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 child 1517 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:28:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 success in 94 seconds
2016-12-07_01:28:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 task output (0 bytes): 
2016-12-07_01:28:08 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 22 done, 0 running, 200 todo
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  collate
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  collated output manifest text to send to API server is 0 bytes with access tokens
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  job output d41d8cd98f00b204e9800998ecf8427e+0
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  running from /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/arvados-cli-0.1.20161017193526/bin/crunch-job with arvados-cli Gem version(s) 0.1.20161017193526, 0.1.20160503204200, 0.1.20151207150126, 0.1.20151023190001
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  check slurm allocation
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  node humgen-05-15 - 8 slots
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  start
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  clean work dirs: start
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk "(index(\\$3, \\"$CRUNCH_TMP\\") == 1){print \\$3}" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']
2016-12-07_01:28:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  clean work dirs: exit 0
2016-12-07_01:28:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  Install docker image 9289c4f1fce1d1dedf9cc7876c33f936+671
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  docker image hash is 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  load docker image: start
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodelist=humgen-05-15','/bin/bash','-o','pipefail','-ec',' if /usr/bin/docker images -q --no-trunc --all | grep -xF 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 >/dev/null; then     exit 0 fi declare -a exit_codes=("${PIPESTATUS[@]}") if [ 0 != "${exit_codes[0]}" ]; then    exit "${exit_codes[0]}"  # `docker images` failed elif [ 1 != "${exit_codes[1]}" ]; then    exit "${exit_codes[1]}"  # `grep` encountered an error else    # Everything worked fine, but grep didn\'t find the image on this host.    arv-get 9289c4f1fce1d1dedf9cc7876c33f936\\+671\\/571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3\\.tar | /usr/bin/docker load fi ']
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  load docker image: exit 0
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  check --memory-swap feature: start
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodes=1','/usr/bin/docker','run','--help']
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  check --memory-swap feature: exit 0
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  check whether default user is UID 0: start
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodes=1','/bin/sh','-ec','/usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21  571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 id --user']
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  check whether default user is UID 0: exit 0
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  Container will run with default user
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  Looking for version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 from repository jr17/hgi
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  Using local repository '/var/lib/arvados/internal.git'
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  Version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 is commit 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  run install script on all workers: start
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio','sh','-c','mkdir -p /data/crunch-tmp/crunch-job/opt && cd /data/crunch-tmp/crunch-job && perl -']
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr Extracting archive: 4b8cf8e1274f8d418cb494d3ea978152
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  run install script on all workers: exit 0
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  script gatk-combinegvcfs.py
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  script_version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  script_parameters {"interval_lists_collection":"4a72f79fd2ed51ac692bc326fb168f37+10043","reference_collection":"1b447d1aa006cde57df03042736750b9+7894","inputs_collection":"b0dd82bb014b6ab8e64968ebdc07aad9+2566847"}
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  runtime_constraints {"keep_cache_mb_per_task":19500,"max_tasks_per_node":8,"min_nodes":1,"docker_image":"mercury/gatk-3.6"}
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  start level 0 with 1 slots
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 0 done, 0 running, 1 todo
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 job_task z8ta6-ot0gb-jcys8e23gw0n907
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 child 5243 started on humgen-05-15.1
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.0.5243','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (1 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jcys8e23gw0n907-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-jcys8e23gw0n907-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jcys8e23gw0n907-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=0 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=0 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-jcys8e23gw0n907 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 0 done, 1 running, 0 todo
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-jcys8e23gw0n907-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jcys8e23gw0n907-0.cid --sig-proxy --memory=250853295k --memory-swap=284407723k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=0 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=0 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-jcys8e23gw0n907 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/cgroup.procs
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/cpuacct.stat
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 0.0000 user 0.0000 sys 0 cpus
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/blkio.io_service_bytes
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:28:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Processing last stream
2016-12-07_01:28:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Finalising stream .
2016-12-07_01:28:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 100_of_200
2016-12-07_01:28:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 100_of_200
2016-12-07_01:28:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 101_of_200
2016-12-07_01:28:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 101_of_200
2016-12-07_01:28:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 102_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 102_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 103_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 103_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 104_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 104_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 105_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 105_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 106_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 106_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 107_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 107_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 108_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 108_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 109_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 109_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 10_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/memory.stat
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 160747520 rss
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/cpuset.cpus
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 4.6400 user 0.4200 sys 32 cpus -- interval 10.0004 seconds 4.6400 user 0.4200 sys
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 259427 tx 3671070 rx
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 10_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 110_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 110_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 111_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 111_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 112_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 112_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 113_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 113_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 114_of_200
2016-12-07_01:28:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 114_of_200
2016-12-07_01:28:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 115_of_200
2016-12-07_01:28:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 115_of_200
2016-12-07_01:28:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 116_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 116_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 117_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 117_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 118_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 118_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 119_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 119_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 11_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 11_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 120_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 120_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 121_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 121_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 122_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 122_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 123_of_200
2016-12-07_01:28:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 123_of_200
2016-12-07_01:28:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 124_of_200
2016-12-07_01:28:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 124_of_200
2016-12-07_01:28:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 125_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 125_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 126_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 126_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 127_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 127_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 128_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 128_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 129_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 129_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 12_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 12_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 130_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 130_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 131_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 131_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 132_of_200
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 177774592 rss
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 8.1800 user 0.6300 sys 32 cpus -- interval 10.0000 seconds 3.5400 user 0.2100 sys
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 856713 tx 4679854 rx -- interval 10.0000 seconds 597286 tx 1008784 rx
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 132_of_200
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 133_of_200
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 133_of_200
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 134_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 134_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 135_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 135_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 136_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 136_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 137_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 137_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 138_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 138_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 139_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 139_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 13_of_200
2016-12-07_01:28:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 13_of_200
2016-12-07_01:28:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 140_of_200
2016-12-07_01:28:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 140_of_200
2016-12-07_01:28:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 141_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 141_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 142_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 142_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 143_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 143_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 144_of_200
2016-12-07_01:28:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 144_of_200
2016-12-07_01:28:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 145_of_200
2016-12-07_01:28:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 145_of_200
2016-12-07_01:28:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 146_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 146_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 147_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 147_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 148_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 148_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 149_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 149_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 14_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 14_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 150_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 150_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 151_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 151_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 152_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 152_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 153_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 153_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 154_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 154_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 155_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 155_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 156_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 190222336 rss
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 11.6600 user 0.7800 sys 32 cpus -- interval 10.0000 seconds 3.4800 user 0.1500 sys
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 1473400 tx 5726521 rx -- interval 10.0000 seconds 616687 tx 1046667 rx
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 156_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 157_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 157_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 158_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 158_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 159_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 159_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 15_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 15_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 160_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 160_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 161_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 161_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 162_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 162_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 163_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 163_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 164_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 164_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 165_of_200
2016-12-07_01:28:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 165_of_200
2016-12-07_01:28:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 166_of_200
2016-12-07_01:28:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 166_of_200
2016-12-07_01:28:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 167_of_200
2016-12-07_01:28:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 167_of_200
2016-12-07_01:28:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 168_of_200
2016-12-07_01:29:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 168_of_200
2016-12-07_01:29:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 169_of_200
2016-12-07_01:29:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 169_of_200
2016-12-07_01:29:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 16_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 16_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 170_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 170_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 171_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 171_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 172_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 214007808 rss
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 13.9400 user 0.9300 sys 32 cpus -- interval 10.0000 seconds 2.2800 user 0.1500 sys
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 1899019 tx 6444281 rx -- interval 10.0000 seconds 425619 tx 717760 rx
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 172_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 173_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 173_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 174_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 174_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 175_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 175_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 176_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 176_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 177_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 177_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 178_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 178_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 179_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 179_of_200
2016-12-07_01:29:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 17_of_200
2016-12-07_01:29:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 17_of_200
2016-12-07_01:29:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 180_of_200
2016-12-07_01:29:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 180_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 181_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 181_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 182_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 182_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 183_of_200
2016-12-07_01:29:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 183_of_200
2016-12-07_01:29:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 184_of_200
2016-12-07_01:29:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 184_of_200
2016-12-07_01:29:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 185_of_200
2016-12-07_01:29:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 185_of_200
2016-12-07_01:29:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 186_of_200
2016-12-07_01:29:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 186_of_200
2016-12-07_01:29:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 187_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 187_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 188_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 188_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 189_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 189_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 18_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 18_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 190_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 190_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 191_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 191_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 192_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 192_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 193_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 193_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 194_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 241840128 rss
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 17.0800 user 1.1000 sys 32 cpus -- interval 10.0000 seconds 3.1400 user 0.1700 sys
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 2469397 tx 7411041 rx -- interval 10.0000 seconds 570378 tx 966760 rx
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 194_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 195_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 195_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 196_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 196_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 197_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 197_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 198_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 198_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 199_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 199_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 19_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 19_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 1_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 1_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 200_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 200_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 20_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 20_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 21_of_200
2016-12-07_01:29:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 21_of_200
2016-12-07_01:29:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 22_of_200
2016-12-07_01:29:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 22_of_200
2016-12-07_01:29:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 23_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 23_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 24_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 24_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 25_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 25_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 26_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 26_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 27_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 27_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 28_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 28_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 29_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 29_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 2_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 2_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 30_of_200
2016-12-07_01:29:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 30_of_200
2016-12-07_01:29:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 31_of_200
2016-12-07_01:29:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 31_of_200
2016-12-07_01:29:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 32_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 32_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 33_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 33_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 34_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 34_of_200
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 270934016 rss
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 20.3600 user 1.2600 sys 32 cpus -- interval 10.0000 seconds 3.2800 user 0.1600 sys
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 3060506 tx 8413678 rx -- interval 10.0000 seconds 591109 tx 1002637 rx
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 35_of_200
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 35_of_200
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 36_of_200
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 36_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 37_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 37_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 38_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 38_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 39_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 39_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 3_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 3_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 40_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 40_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 41_of_200
2016-12-07_01:29:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 41_of_200
2016-12-07_01:29:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 42_of_200
2016-12-07_01:29:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 42_of_200
2016-12-07_01:29:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 43_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 43_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 44_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 44_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 45_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 45_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 46_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 46_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 47_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 47_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 48_of_200
2016-12-07_01:29:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 48_of_200
2016-12-07_01:29:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 49_of_200
2016-12-07_01:29:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 49_of_200
2016-12-07_01:29:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 4_of_200
2016-12-07_01:29:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 4_of_200
2016-12-07_01:29:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 50_of_200
2016-12-07_01:29:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 50_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 51_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 51_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 52_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 52_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 53_of_200
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 53_of_200
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 54_of_200
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 54_of_200
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 55_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 298774528 rss
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 23.3200 user 1.4200 sys 32 cpus -- interval 10.0000 seconds 2.9600 user 0.1600 sys
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 3583227 tx 9307377 rx -- interval 10.0000 seconds 522721 tx 893699 rx
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 55_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 56_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 56_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 57_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 57_of_200
2016-12-07_01:29:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 58_of_200
2016-12-07_01:29:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 58_of_200
2016-12-07_01:29:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 59_of_200
2016-12-07_01:29:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 59_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 5_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 5_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 60_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 60_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 61_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 61_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 62_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 62_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 63_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 63_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 64_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 64_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 65_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 65_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 66_of_200
2016-12-07_01:29:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 66_of_200
2016-12-07_01:29:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 67_of_200
2016-12-07_01:29:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 67_of_200
2016-12-07_01:29:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 68_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 68_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 69_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 69_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 6_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 6_of_200
2016-12-07_01:29:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 70_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 70_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 71_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 71_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 72_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 72_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 73_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 73_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 74_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 74_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 75_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 309661696 rss
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 26.8400 user 1.7000 sys 32 cpus -- interval 9.9999 seconds 3.5200 user 0.2800 sys
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 4141127 tx 10205868 rx -- interval 9.9999 seconds 557900 tx 898491 rx
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 75_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 76_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 76_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 77_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 77_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 78_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 78_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 79_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 79_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 7_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 7_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 80_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 80_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 81_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 81_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 82_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 82_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 83_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 83_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 84_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 84_of_200
2016-12-07_01:29:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 85_of_200
2016-12-07_01:29:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 85_of_200
2016-12-07_01:29:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 86_of_200
2016-12-07_01:29:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 86_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 87_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 87_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 88_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 88_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 89_of_200
2016-12-07_01:29:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 89_of_200
2016-12-07_01:29:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 8_of_200
2016-12-07_01:29:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 8_of_200
2016-12-07_01:29:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 90_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 90_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 91_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 91_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 92_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 92_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 93_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 93_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 94_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 94_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 95_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 95_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 96_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 96_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 97_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 97_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 98_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 309698560 rss
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 30.0400 user 1.8600 sys 32 cpus -- interval 10.0000 seconds 3.2000 user 0.1600 sys
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 4712642 tx 11203923 rx -- interval 10.0000 seconds 571515 tx 998055 rx
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 98_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 99_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 99_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 9_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 9_of_200
2016-12-07_01:29:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Ending task 0 successfully
2016-12-07_01:29:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 child 5243 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:29:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 success in 92 seconds
2016-12-07_01:29:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 task output (0 bytes): 
2016-12-07_01:29:55 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:04 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 200 new tasks from API server.
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 1 done, 0 running, 200 todo
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  wait for last 0 children to finish
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  start level 1 with 8 slots
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 1 done, 0 running, 200 todo
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 job_task z8ta6-ot0gb-hizpczdtbjel8pt
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 child 8311 started on humgen-05-15.1
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 job_task z8ta6-ot0gb-t8qg32goo9pc50a
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 child 8322 started on humgen-05-15.2
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 job_task z8ta6-ot0gb-gaupyyeay9of4sb
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 child 8327 started on humgen-05-15.3
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 job_task z8ta6-ot0gb-9d2ze0wl27bbr45
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 child 8335 started on humgen-05-15.4
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 job_task z8ta6-ot0gb-iyvt2bz6ed4h7sk
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 child 8339 started on humgen-05-15.5
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 job_task z8ta6-ot0gb-5h8cpstnmzy5a9s
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 child 8353 started on humgen-05-15.6
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 job_task z8ta6-ot0gb-qw3ghu26grvq3m3
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 child 8357 started on humgen-05-15.7
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 job_task z8ta6-ot0gb-2q378ahzc2xpimi
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 child 8361 started on humgen-05-15.8
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.1.8311','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-hizpczdtbjel8pt-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-hizpczdtbjel8pt-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-hizpczdtbjel8pt-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=1 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-hizpczdtbjel8pt --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-hizpczdtbjel8pt-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-hizpczdtbjel8pt-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=1 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-hizpczdtbjel8pt --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/cgroup.procs
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/memory.stat
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 2867200 rss
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/cpuacct.stat
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/cpuset.cpus
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/blkio.io_service_bytes
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.2.8322','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t8qg32goo9pc50a-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-t8qg32goo9pc50a-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t8qg32goo9pc50a-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=2 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-t8qg32goo9pc50a --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.3.8327','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-gaupyyeay9of4sb-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-gaupyyeay9of4sb-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-gaupyyeay9of4sb-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=3 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-gaupyyeay9of4sb --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.4.8335','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9d2ze0wl27bbr45-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-9d2ze0wl27bbr45-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9d2ze0wl27bbr45-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=4 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-9d2ze0wl27bbr45 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.5.8339','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-iyvt2bz6ed4h7sk-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-iyvt2bz6ed4h7sk-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-iyvt2bz6ed4h7sk-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=5 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-iyvt2bz6ed4h7sk --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.6.8353','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5h8cpstnmzy5a9s-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-5h8cpstnmzy5a9s-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5h8cpstnmzy5a9s-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=6 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-5h8cpstnmzy5a9s --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.7.8357','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qw3ghu26grvq3m3-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qw3ghu26grvq3m3-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qw3ghu26grvq3m3-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=7 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-qw3ghu26grvq3m3 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.8.8361','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2q378ahzc2xpimi-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2q378ahzc2xpimi-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2q378ahzc2xpimi-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=8 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2q378ahzc2xpimi --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 1 done, 8 running, 192 todo
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Interval_List collection mounted at /keep/812316f6c95d8641412912523842bbfc+94
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Total chunk length is 15618872
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Have 1 intervals
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: blkio:0:0 0 write 12523 read -- interval 10.0000 seconds 0 write 12523 read
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Have 5400 potential reusable task outputs
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Have 5400 potentially reusable tasks
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Creating task to process interval: [9:6148617-21767488]
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr No reusable JobTask matched key parameters [u'100_of_200', u'a241297995deeab501f9bd99dc533315+14700', '9:6148617-21767488', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 48857088 rss
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: cpu 0.7000 user 0.2100 sys 32 cpus -- interval 10.0001 seconds 0.6800 user 0.2100 sys
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: net:eth0 35651 tx 3238200 rx
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Ending task 1 successfully
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 child 8311 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 success in 11 seconds
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 task output (0 bytes): 
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 2 done, 7 running, 193 todo
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 job_task z8ta6-ot0gb-0rkh929u4mx4kvh
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 child 9178 started on humgen-05-15.1
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.9.9178','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0rkh929u4mx4kvh-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0rkh929u4mx4kvh-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0rkh929u4mx4kvh-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=9 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-0rkh929u4mx4kvh --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-t8qg32goo9pc50a-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t8qg32goo9pc50a-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=2 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-t8qg32goo9pc50a --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/cgroup.procs
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/memory.stat
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 3014656 rss
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/cpuacct.stat
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/cpuset.cpus
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/blkio.io_service_bytes
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Interval_List collection mounted at /keep/6af141a235ccb024f6a32fc12ee98dcf+94
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Total chunk length is 15618872
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Have 1 intervals
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 2 done, 8 running, 192 todo
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24092672 rss
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: cpu 0.4900 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.4700 user 0.1800 sys
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: net:eth0 21807 tx 238280 rx
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Have 5400 potential reusable task outputs
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Have 5400 potentially reusable tasks
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Creating task to process interval: [9:21767489-37386360]
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr No reusable JobTask matched key parameters [u'101_of_200', u'1b848222e216cbae87e6932d5820a285+14700', '9:21767489-37386360', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Ending task 1 successfully
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 child 8322 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 success in 23 seconds
2016-12-07_01:30:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 task output (0 bytes): 
2016-12-07_01:30:29 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 3 done, 7 running, 193 todo
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 job_task z8ta6-ot0gb-04fu0g2l0893trg
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 child 9506 started on humgen-05-15.2
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.10.9506','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-04fu0g2l0893trg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-04fu0g2l0893trg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-04fu0g2l0893trg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=10 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-04fu0g2l0893trg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr srun: Job step created
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-gaupyyeay9of4sb-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-gaupyyeay9of4sb-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=3 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-gaupyyeay9of4sb --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d78569bda201583317540e52c3114637b00b0246e0874874cd3f0d0c0888e076/cgroup.procs
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d78569bda201583317540e52c3114637b00b0246e0874874cd3f0d0c0888e076/cpuacct.stat
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: cpu 0.0100 user 0.0000 sys 0 cpus
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d78569bda201583317540e52c3114637b00b0246e0874874cd3f0d0c0888e076/blkio.io_service_bytes
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Interval_List collection mounted at /keep/2e7cf8719961c4cc5b2ffdd92ff99dd0+94
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Total chunk length is 15618872
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Have 1 intervals
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:36 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 3 done, 8 running, 192 todo
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Have 5400 potential reusable task outputs
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Have 5400 potentially reusable tasks
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Creating task to process interval: [9:37386361-53005232]
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr No reusable JobTask matched key parameters [u'102_of_200', u'0b01f4aa638023c7844e14aa840fe4cc+14532', '9:37386361-53005232', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Ending task 1 successfully
2016-12-07_01:30:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 child 8327 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:30:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 success in 34 seconds
2016-12-07_01:30:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 task output (0 bytes): 
2016-12-07_01:30:39 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:45 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 4 done, 7 running, 193 todo
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 job_task z8ta6-ot0gb-tgduaa5l7ohexgw
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 child 9860 started on humgen-05-15.3
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.11.9860','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-tgduaa5l7ohexgw-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-tgduaa5l7ohexgw-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-tgduaa5l7ohexgw-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=11 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-tgduaa5l7ohexgw --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr srun: Job step created
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-iyvt2bz6ed4h7sk-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-iyvt2bz6ed4h7sk-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=5 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-iyvt2bz6ed4h7sk --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/cgroup.procs
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/memory.stat
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4927488 rss
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/cpuacct.stat
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/cpuset.cpus
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/blkio.io_service_bytes
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Interval_List collection mounted at /keep/61144aa3079bc748eb474251bb12d2af+94
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Total chunk length is 15618872
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Have 1 intervals
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 4 done, 8 running, 192 todo
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Have 5400 potential reusable task outputs
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46034944 rss
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: cpu 0.6800 user 0.2000 sys 32 cpus -- interval 10.0002 seconds 0.6600 user 0.2000 sys
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: net:eth0 34323 tx 3236409 rx
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Have 5400 potentially reusable tasks
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Creating task to process interval: [9:68624105-84242976]
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr No reusable JobTask matched key parameters [u'104_of_200', u'0e7a8a6d12ed4d9a7f7a47c972f98c08+14700', '9:68624105-84242976', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Ending task 1 successfully
2016-12-07_01:30:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 child 8339 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:30:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 success in 45 seconds
2016-12-07_01:30:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 task output (0 bytes): 
2016-12-07_01:30:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 5 done, 7 running, 193 todo
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 job_task z8ta6-ot0gb-t2losx80lggglyb
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 child 10215 started on humgen-05-15.5
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr srun: Job step created
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0rkh929u4mx4kvh-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0rkh929u4mx4kvh-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=9 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-0rkh929u4mx4kvh --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/cgroup.procs
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/memory.stat
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4640768 rss
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/cpuacct.stat
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/cpuset.cpus
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/blkio.io_service_bytes
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Interval_List collection mounted at /keep/64696f9b403b7674c728386882cbd3b7+94
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Total chunk length is 15498872
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Have 1 intervals
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.12.10215','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t2losx80lggglyb-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-t2losx80lggglyb-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t2losx80lggglyb-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=12 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-t2losx80lggglyb --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 5 done, 8 running, 192 todo
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: blkio:0:0 0 write 12564 read -- interval 10.0000 seconds 0 write 12564 read
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Creating task to process interval: [9:131099593-141213431 10:1-5385033]
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr No reusable JobTask matched key parameters [u'108_of_200', u'7e64dee294f55ab4bfb56c1a154618f8+14704', '9:131099593-141213431 10:1-5385033', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 49082368 rss
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: cpu 0.7600 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.7200 user 0.1700 sys
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: net:eth0 36730 tx 3244582 rx
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Ending task 1 successfully
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr srun: Job step created
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 child 9178 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 success in 38 seconds
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 task output (0 bytes): 
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 6 done, 7 running, 193 todo
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 job_task z8ta6-ot0gb-6u2qax3rliz18jb
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 child 10526 started on humgen-05-15.1
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.13.10526','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-6u2qax3rliz18jb-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-6u2qax3rliz18jb-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-6u2qax3rliz18jb-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=13 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-6u2qax3rliz18jb --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-5h8cpstnmzy5a9s-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5h8cpstnmzy5a9s-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=6 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-5h8cpstnmzy5a9s --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/cgroup.procs
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/memory.stat
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5849088 rss
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/cpuacct.stat
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/cpuset.cpus
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: cpu 0.0200 user 0.0200 sys 32 cpus
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/blkio.io_service_bytes
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Interval_List collection mounted at /keep/4325c7dac5c39f2db578e849695a12aa+94
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Total chunk length is 15618872
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Have 1 intervals
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 6 done, 8 running, 192 todo
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Creating task to process interval: [9:84242977-99861848]
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr No reusable JobTask matched key parameters [u'105_of_200', u'3a3bb24e84ba30a06b3d46caa3b1c040+14700', '9:84242977-99861848', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Ending task 1 successfully
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr srun: Job step created
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 child 8353 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 success in 65 seconds
2016-12-07_01:31:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 task output (0 bytes): 
2016-12-07_01:31:11 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 notice: task is not in slurm queue but srun process 8361 has not exited
2016-12-07_01:31:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 notice: task is not in slurm queue but srun process 8335 has not exited
2016-12-07_01:31:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 notice: task is not in slurm queue but srun process 8357 has not exited
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 7 done, 7 running, 193 todo
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 job_task z8ta6-ot0gb-65d5grcm0p15z7j
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 child 10870 started on humgen-05-15.6
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-t2losx80lggglyb-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t2losx80lggglyb-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=12 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-t2losx80lggglyb --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/cgroup.procs
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/memory.stat
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 8159232 rss
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/cpuacct.stat
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/cpuset.cpus
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: cpu 0.0500 user 0.0000 sys 32 cpus
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/blkio.io_service_bytes
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Interval_List collection mounted at /keep/11837032d32bcfff7055a484f9c0ddb8+94
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Total chunk length is 15618872
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Have 1 intervals
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.14.10870','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-65d5grcm0p15z7j-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-65d5grcm0p15z7j-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-65d5grcm0p15z7j-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=14 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-65d5grcm0p15z7j --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 7 done, 8 running, 192 todo
2016-12-07_01:31:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Creating task to process interval: [10:21003906-36622777]
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr No reusable JobTask matched key parameters [u'110_of_200', u'e20f5f00b25cd7e53f799770dbe9e308+14700', '10:21003906-36622777', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Ending task 1 successfully
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid 40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b, parent docker, root /sys/fs/cgroup
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid 40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b, parent docker, root /sys/fs/cgroup
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid 40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b, parent docker, root /sys/fs/cgroup
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid 40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b, parent docker, root /sys/fs/cgroup
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 child 10215 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 success in 24 seconds
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 task output (0 bytes): 
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 8 done, 7 running, 193 todo
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 job_task z8ta6-ot0gb-w4g13gcmycpxkrd
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 child 11199 started on humgen-05-15.5
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.15.11199','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w4g13gcmycpxkrd-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-w4g13gcmycpxkrd-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w4g13gcmycpxkrd-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=15 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-w4g13gcmycpxkrd --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr srun: Job step created
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qw3ghu26grvq3m3-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qw3ghu26grvq3m3-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=7 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-qw3ghu26grvq3m3 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/cgroup.procs
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/memory.stat
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4849664 rss
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/cpuacct.stat
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/cpuset.cpus
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/blkio.io_service_bytes
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Interval_List collection mounted at /keep/34f0119dab554faf0acfc04f3cae7c1d+94
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Total chunk length is 15618872
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Have 1 intervals
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 8 done, 8 running, 192 todo
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: cpu 0.5100 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.4700 user 0.1500 sys
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: net:eth0 21877 tx 238215 rx
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Creating task to process interval: [9:99861849-115480720]
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr No reusable JobTask matched key parameters [u'106_of_200', u'dc8fa207ac10a02e00e1ba8ab1687ad9+14700', '9:99861849-115480720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Ending task 1 successfully
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 child 8357 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 success in 87 seconds
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 task output (0 bytes): 
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 9 done, 7 running, 193 todo
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 job_task z8ta6-ot0gb-mrupikb9g03ehny
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 child 11540 started on humgen-05-15.7
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr srun: Job step created
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-6u2qax3rliz18jb-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-6u2qax3rliz18jb-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=13 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-6u2qax3rliz18jb --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/cgroup.procs
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/memory.stat
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5693440 rss
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/cpuacct.stat
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/cpuset.cpus
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: cpu 0.0500 user 0.0000 sys 32 cpus
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/blkio.io_service_bytes
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Interval_List collection mounted at /keep/d6cf1afdecfe2ae2261292a571a3759b+94
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Total chunk length is 15618872
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Have 1 intervals
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.16.11540','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-mrupikb9g03ehny-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-mrupikb9g03ehny-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-mrupikb9g03ehny-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=16 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-mrupikb9g03ehny --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 9 done, 8 running, 192 todo
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Creating task to process interval: [10:36622778-52241649]
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr No reusable JobTask matched key parameters [u'111_of_200', u'94eedabc31fa41e0ae0f2bed3edca749+14699', '10:36622778-52241649', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Ending task 1 successfully
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 child 10526 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 success in 35 seconds
2016-12-07_01:31:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 task output (0 bytes): 
2016-12-07_01:31:44 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 killing orphaned srun process 8361 (task not in slurm queue, no stderr received in last 86s)
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 sending 2x signal 2 to pid 8361
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 killing orphaned srun process 8335 (task not in slurm queue, no stderr received in last 86s)
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 sending 2x signal 2 to pid 8335
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 notice: task is not in slurm queue but srun process 9506 has not exited
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 10 done, 7 running, 193 todo
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 job_task z8ta6-ot0gb-xn87jpqo3q9dql1
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 child 12176 started on humgen-05-15.1
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.17.12176','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xn87jpqo3q9dql1-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xn87jpqo3q9dql1-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xn87jpqo3q9dql1-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=17 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-xn87jpqo3q9dql1 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr srun: Job step created
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-tgduaa5l7ohexgw-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-tgduaa5l7ohexgw-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=11 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-tgduaa5l7ohexgw --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/cgroup.procs
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/memory.stat
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4898816 rss
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/cpuacct.stat
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/cpuset.cpus
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/blkio.io_service_bytes
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Interval_List collection mounted at /keep/47924fc036d6a931b5f0cf53440e0969+92
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Total chunk length is 15618872
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Have 1 intervals
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 child 8335 on humgen-05-15.4 exit 1 success=
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:31:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 failure (#1, temporary) after 106 seconds
2016-12-07_01:31:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 task output (0 bytes): 
2016-12-07_01:31:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 child 8361 on humgen-05-15.8 exit 1 success=
2016-12-07_01:31:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:31:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 failure (#1, temporary) after 107 seconds
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 task output (0 bytes): 
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 child 9860 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 success in 69 seconds
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 task output (0 bytes): 
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970  3 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 11 done, 5 running, 195 todo
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 job_task z8ta6-ot0gb-ljma8zh3j9drwfa
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 child 12495 started on humgen-05-15.4
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 job_task z8ta6-ot0gb-zex553q8rtr26xg
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 child 12499 started on humgen-05-15.8
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 job_task z8ta6-ot0gb-9pmmsjzn1rgx4qz
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 child 12503 started on humgen-05-15.3
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.18.12495','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ljma8zh3j9drwfa-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ljma8zh3j9drwfa-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ljma8zh3j9drwfa-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=18 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ljma8zh3j9drwfa --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.19.12499','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zex553q8rtr26xg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zex553q8rtr26xg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zex553q8rtr26xg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=19 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-zex553q8rtr26xg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr srun: Job step created
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-w4g13gcmycpxkrd-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w4g13gcmycpxkrd-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=15 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-w4g13gcmycpxkrd --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/cgroup.procs
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/memory.stat
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 1626112 rss
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/cpuacct.stat
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/cpuset.cpus
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/blkio.io_service_bytes
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Interval_List collection mounted at /keep/f73eab8bfca061b956c4cbaa59d62ccd+94
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Total chunk length is 15618872
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Have 1 intervals
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.20.12503','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9pmmsjzn1rgx4qz-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-9pmmsjzn1rgx4qz-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9pmmsjzn1rgx4qz-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=20 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-9pmmsjzn1rgx4qz --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 11 done, 8 running, 192 todo
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:32:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:32:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: cpu 0.5200 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.5000 user 0.1500 sys
2016-12-07_01:32:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: net:eth0 21877 tx 238084 rx
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Creating task to process interval: [10:67860522-83479393]
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr No reusable JobTask matched key parameters [u'113_of_200', u'45d9d4c732c4ecae90a7f5ec11993512+14700', '10:67860522-83479393', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Ending task 1 successfully
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 child 11199 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 success in 39 seconds
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 task output (0 bytes): 
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 12 done, 7 running, 193 todo
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 job_task z8ta6-ot0gb-zi4wk7zwhebjuoo
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 child 12866 started on humgen-05-15.5
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.21.12866','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zi4wk7zwhebjuoo-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zi4wk7zwhebjuoo-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zi4wk7zwhebjuoo-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=21 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-zi4wk7zwhebjuoo --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr srun: Job step created
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-65d5grcm0p15z7j-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-65d5grcm0p15z7j-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=14 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-65d5grcm0p15z7j --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/cgroup.procs
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/memory.stat
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7585792 rss
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/cpuacct.stat
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/cpuset.cpus
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/blkio.io_service_bytes
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Interval_List collection mounted at /keep/fe3ded28e7bf89d25f8d4e08ecc0d6c5+94
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Total chunk length is 15618872
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Have 1 intervals
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 12 done, 8 running, 192 todo
2016-12-07_01:32:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Creating task to process interval: [10:52241650-67860521]
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr No reusable JobTask matched key parameters [u'112_of_200', u'9e1996fed17b6a2fca279a862266ce9d+14700', '10:52241650-67860521', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Ending task 1 successfully
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 47923200 rss
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: cpu 0.7700 user 0.2100 sys 32 cpus -- interval 10.0002 seconds 0.7400 user 0.2100 sys
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: net:eth0 39639 tx 3252569 rx
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 child 10870 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 success in 60 seconds
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 task output (0 bytes): 
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 killing orphaned srun process 9506 (task not in slurm queue, no stderr received in last 99s)
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 sending 2x signal 2 to pid 9506
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 13 done, 7 running, 193 todo
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 job_task z8ta6-ot0gb-blcloqm3400c4bg
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 child 13267 started on humgen-05-15.6
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zex553q8rtr26xg-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zex553q8rtr26xg-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=19 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-zex553q8rtr26xg --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/cgroup.procs
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/memory.stat
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5861376 rss
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/cpuacct.stat
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/cpuset.cpus
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/blkio.io_service_bytes
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Interval_List collection mounted at /keep/76a230c91cc5d0053b75306d58906b2a+94
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Total chunk length is 15498872
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Have 1 intervals
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.22.13267','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-blcloqm3400c4bg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-blcloqm3400c4bg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-blcloqm3400c4bg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=22 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-blcloqm3400c4bg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 child 9506 on humgen-05-15.2 exit 1 success=
2016-12-07_01:32:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:32:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 failure (#1, temporary) after 111 seconds
2016-12-07_01:32:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 task output (0 bytes): 
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 13 done, 7 running, 193 todo
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Creating task to process interval: [10:130336010-135534747 11:1-10300134]
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr No reusable JobTask matched key parameters [u'117_of_200', u'a986cd76d508917b0b96bd60c6107961+14704', '10:130336010-135534747 11:1-10300134', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Ending task 1 successfully
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: blkio:0:0 0 write 12567 read -- interval 10.0000 seconds 0 write 12567 read
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 child 12499 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 success in 27 seconds
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 task output (0 bytes): 
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 14 done, 6 running, 194 todo
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 job_task z8ta6-ot0gb-cqpt669hl2r0t18
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 child 13584 started on humgen-05-15.8
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 job_task z8ta6-ot0gb-xo8sk9whc1n6eon
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 child 13597 started on humgen-05-15.2
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.23.13584','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-cqpt669hl2r0t18-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-cqpt669hl2r0t18-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-cqpt669hl2r0t18-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=23 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-cqpt669hl2r0t18 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.24.13597','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xo8sk9whc1n6eon-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xo8sk9whc1n6eon-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xo8sk9whc1n6eon-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=24 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-xo8sk9whc1n6eon --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr srun: Job step created
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-mrupikb9g03ehny-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-mrupikb9g03ehny-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=16 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-mrupikb9g03ehny --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/cgroup.procs
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/memory.stat
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: mem 86016 cache 0 swap 0 pgmajfault 5816320 rss
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/cpuacct.stat
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/cpuset.cpus
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: cpu 0.0500 user 0.0200 sys 32 cpus
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/blkio.io_service_bytes
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Interval_List collection mounted at /keep/8b0a6de6be0a7b3dd7360336477e02fd+94
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Total chunk length is 15618872
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Have 1 intervals
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 14 done, 8 running, 192 todo
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:32:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24092672 rss
2016-12-07_01:32:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: cpu 0.5000 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4500 user 0.1400 sys
2016-12-07_01:32:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: net:eth0 21877 tx 238414 rx -- interval 10.0001 seconds 21787 tx 238324 rx
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Creating task to process interval: [10:83479394-99098265]
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr No reusable JobTask matched key parameters [u'114_of_200', u'c3bc3b07edcf5d2b7b9bfb2b5154d56f+14700', '10:83479394-99098265', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Ending task 1 successfully
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr srun: Job step created
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 child 11540 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 success in 63 seconds
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 task output (0 bytes): 
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 15 done, 7 running, 193 todo
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 job_task z8ta6-ot0gb-ltvbgm6zwzhs2or
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 child 13988 started on humgen-05-15.7
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ljma8zh3j9drwfa-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ljma8zh3j9drwfa-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=18 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ljma8zh3j9drwfa --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/cgroup.procs
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/memory.stat
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4964352 rss
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/cpuacct.stat
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/cpuset.cpus
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/blkio.io_service_bytes
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Interval_List collection mounted at /keep/359a96b521b56e8d8e36b57450fa80ac+94
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Total chunk length is 15618872
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Have 1 intervals
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.25.13988','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ltvbgm6zwzhs2or-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ltvbgm6zwzhs2or-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ltvbgm6zwzhs2or-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=25 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ltvbgm6zwzhs2or --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 15 done, 8 running, 192 todo
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Creating task to process interval: [10:114717138-130336009]
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr No reusable JobTask matched key parameters [u'116_of_200', u'6365525c0ee094fee362794dbbab6a30+14700', '10:114717138-130336009', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Ending task 1 successfully
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 child 12495 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 success in 51 seconds
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 task output (0 bytes): 
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 notice: task is not in slurm queue but srun process 12176 has not exited
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 16 done, 7 running, 193 todo
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 job_task z8ta6-ot0gb-dym505mzgyoooiy
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 child 14344 started on humgen-05-15.4
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-cqpt669hl2r0t18-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-cqpt669hl2r0t18-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=23 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-cqpt669hl2r0t18 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/cgroup.procs
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/memory.stat
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: mem 94208 cache 0 swap 0 pgmajfault 5206016 rss
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/cpuacct.stat
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/cpuset.cpus
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/blkio.io_service_bytes
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Interval_List collection mounted at /keep/b34c79e916b3f4868c677e831b11c26d+94
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Total chunk length is 15618872
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Have 1 intervals
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.26.14344','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dym505mzgyoooiy-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-dym505mzgyoooiy-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dym505mzgyoooiy-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=26 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-dym505mzgyoooiy --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:01 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 16 done, 8 running, 192 todo
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Creating task to process interval: [11:41537879-57156750]
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr No reusable JobTask matched key parameters [u'120_of_200', u'4253ced78675bc039ec63c878e7f8d99+14700', '11:41537879-57156750', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Ending task 1 successfully
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr srun: Job step created
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 child 13584 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 success in 27 seconds
2016-12-07_01:33:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 task output (0 bytes): 
2016-12-07_01:33:04 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 17 done, 7 running, 193 todo
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 job_task z8ta6-ot0gb-62gm4qd6vfzcrvk
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 child 14644 started on humgen-05-15.8
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.27.14644','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-62gm4qd6vfzcrvk-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-62gm4qd6vfzcrvk-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-62gm4qd6vfzcrvk-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=27 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-62gm4qd6vfzcrvk --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-9pmmsjzn1rgx4qz-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9pmmsjzn1rgx4qz-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=20 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-9pmmsjzn1rgx4qz --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/cgroup.procs
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/memory.stat
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: mem 16384 cache 0 swap 0 pgmajfault 4317184 rss
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/cpuacct.stat
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/cpuset.cpus
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: cpu 0.0500 user 0.0000 sys 32 cpus
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/blkio.io_service_bytes
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: net:eth0 0 tx 0 rx
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Interval_List collection mounted at /keep/d9aa702cd7b3a1e6b6dec54b4a3d8e35+94
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Total chunk length is 15618872
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Have 1 intervals
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 17 done, 8 running, 192 todo
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Creating task to process interval: [11:10300135-25919006]
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr No reusable JobTask matched key parameters [u'118_of_200', u'79b22e2a5c2a107fd090c78ed23ff2e7+14700', '11:10300135-25919006', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Ending task 1 successfully
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 child 12503 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 success in 71 seconds
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 task output (0 bytes): 
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 notice: task is not in slurm queue but srun process 12866 has not exited
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 18 done, 7 running, 193 todo
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 job_task z8ta6-ot0gb-f5vx8p9vrdy4mpx
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 child 14959 started on humgen-05-15.3
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-dym505mzgyoooiy-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dym505mzgyoooiy-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=26 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-dym505mzgyoooiy --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/cgroup.procs
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/memory.stat
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5750784 rss
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/cpuacct.stat
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/cpuset.cpus
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/blkio.io_service_bytes
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Interval_List collection mounted at /keep/ddb0ad35fd81534c2371a454b4e54268+94
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Total chunk length is 15618872
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Have 1 intervals
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.28.14959','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-f5vx8p9vrdy4mpx-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-f5vx8p9vrdy4mpx-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-f5vx8p9vrdy4mpx-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=28 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-f5vx8p9vrdy4mpx --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 18 done, 8 running, 192 todo
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Creating task to process interval: [11:88394495-104013366]
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr No reusable JobTask matched key parameters [u'123_of_200', u'b28694acffb44b65c84fe09c422e709c+14700', '11:88394495-104013366', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Ending task 1 successfully
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr srun: Job step created
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 child 14344 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 success in 23 seconds
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 task output (0 bytes): 
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:29 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 19 done, 7 running, 193 todo
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 job_task z8ta6-ot0gb-qdl6sjf1fvebcfg
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 child 15286 started on humgen-05-15.4
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xn87jpqo3q9dql1-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xn87jpqo3q9dql1-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=17 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-xn87jpqo3q9dql1 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/cgroup.procs
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/memory.stat
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5836800 rss
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/cpuacct.stat
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/cpuset.cpus
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/blkio.io_service_bytes
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Interval_List collection mounted at /keep/e05c9a56f87bbb0e87bd78edfb2138bb+94
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Total chunk length is 15618872
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Have 1 intervals
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.29.15286','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qdl6sjf1fvebcfg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qdl6sjf1fvebcfg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qdl6sjf1fvebcfg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=29 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-qdl6sjf1fvebcfg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 19 done, 8 running, 192 todo
2016-12-07_01:33:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Creating task to process interval: [10:99098266-114717137]
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr No reusable JobTask matched key parameters [u'115_of_200', u'5601c60d4022448c1a998597fc7d32dc+14700', '10:99098266-114717137', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Ending task 1 successfully
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr srun: Job step created
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 child 12176 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 success in 103 seconds
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 task output (0 bytes): 
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 notice: task is not in slurm queue but srun process 13597 has not exited
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 notice: task is not in slurm queue but srun process 13267 has not exited
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 20 done, 7 running, 193 todo
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 job_task z8ta6-ot0gb-4ojxt1g1nb8k256
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 child 15891 started on humgen-05-15.1
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.30.15891','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-4ojxt1g1nb8k256-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-4ojxt1g1nb8k256-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-4ojxt1g1nb8k256-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=30 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-4ojxt1g1nb8k256 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zi4wk7zwhebjuoo-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zi4wk7zwhebjuoo-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=21 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-zi4wk7zwhebjuoo --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/cgroup.procs
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/memory.stat
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5791744 rss
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/cpuacct.stat
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/cpuset.cpus
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/blkio.io_service_bytes
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: net:eth0 0 tx 0 rx
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Interval_List collection mounted at /keep/dbba1077cec452fe00018827674a6082+94
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Total chunk length is 15618872
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Have 1 intervals
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:41 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 20 done, 8 running, 192 todo
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24293376 rss
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: cpu 0.5300 user 0.1500 sys 32 cpus -- interval 10.0002 seconds 0.4900 user 0.1400 sys
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: net:eth0 26497 tx 3078455 rx -- interval 10.0002 seconds 26497 tx 3078455 rx
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Creating task to process interval: [11:25919007-41537878]
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr No reusable JobTask matched key parameters [u'119_of_200', u'e715710b28adf5479c2b4ee6a270292a+14700', '11:25919007-41537878', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Ending task 1 successfully
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr srun: Job step created
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 child 12866 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 success in 91 seconds
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 task output (0 bytes): 
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 21 done, 7 running, 193 todo
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 job_task z8ta6-ot0gb-d7p9u5ff3wkrs4o
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 child 16264 started on humgen-05-15.5
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.31.16264','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-d7p9u5ff3wkrs4o-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-d7p9u5ff3wkrs4o-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-d7p9u5ff3wkrs4o-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=31 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-d7p9u5ff3wkrs4o --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-blcloqm3400c4bg-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-blcloqm3400c4bg-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=22 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-blcloqm3400c4bg --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/cgroup.procs
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/memory.stat
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 6094848 rss
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/cpuacct.stat
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/cpuset.cpus
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/blkio.io_service_bytes
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Interval_List collection mounted at /keep/a19791935337560ca00961b9d0c50277+92
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Total chunk length is 15618872
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Have 1 intervals
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 21 done, 8 running, 192 todo
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 notice: task is not in slurm queue but srun process 13988 has not exited
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:33:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:33:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: cpu 0.4900 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4500 user 0.1600 sys
2016-12-07_01:33:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: net:eth0 22183 tx 238403 rx
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Creating task to process interval: [1:156188721-171807592]
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr No reusable JobTask matched key parameters [u'11_of_200', u'0407d47186859c17195bd2efb49797bb+14616', '1:156188721-171807592', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Ending task 1 successfully
2016-12-07_01:33:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 child 13267 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:33:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 success in 93 seconds
2016-12-07_01:33:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 task output (0 bytes): 
2016-12-07_01:33:58 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:34:04 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 22 done, 7 running, 193 todo
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 job_task z8ta6-ot0gb-0y41phqbnenlsfs
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 child 16643 started on humgen-05-15.6
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.32.16643','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0y41phqbnenlsfs-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0y41phqbnenlsfs-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0y41phqbnenlsfs-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=32 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-0y41phqbnenlsfs --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr srun: Job step created
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ltvbgm6zwzhs2or-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ltvbgm6zwzhs2or-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=25 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ltvbgm6zwzhs2or --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/cgroup.procs
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/memory.stat
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5177344 rss
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/cpuacct.stat
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/cpuset.cpus
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/blkio.io_service_bytes
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Mounting task input collection to get interval_list
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Interval_List collection mounted at /keep/bbfdffd38f1432cef552ced110ab3a90+94
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Total chunk length is 15618872
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Have 1 intervals
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 22 done, 8 running, 192 todo
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Have 5400 potential reusable task outputs
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Have 5400 potentially reusable tasks
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Creating task to process interval: [11:72775623-88394494]
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr No reusable JobTask matched key parameters [u'122_of_200', u'af62e8edc576e4baf02413c2134c9270+14700', '11:72775623-88394494', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Ending task 1 successfully
2016-12-07_01:34:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 child 13988 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:34:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 success in 78 seconds
2016-12-07_01:34:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 task output (0 bytes): 
2016-12-07_01:34:08 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 killing orphaned srun process 13597 (task not in slurm queue, no stderr received in last 74s)
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 sending 2x signal 2 to pid 13597
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 23 done, 7 running, 193 todo
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 job_task z8ta6-ot0gb-r9pvowy0tacjusv
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 child 16999 started on humgen-05-15.7
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr srun: Job step created
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-62gm4qd6vfzcrvk-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-62gm4qd6vfzcrvk-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=27 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-62gm4qd6vfzcrvk --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/cgroup.procs
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/memory.stat
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5271552 rss
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/cpuacct.stat
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/cpuset.cpus
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: cpu 0.0200 user 0.0100 sys 32 cpus
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/blkio.io_service_bytes
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Mounting task input collection to get interval_list
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Interval_List collection mounted at /keep/b4389106228a3052a1940c08eb93c60e+94
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Total chunk length is 15618872
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Have 1 intervals
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.33.16999','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-r9pvowy0tacjusv-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-r9pvowy0tacjusv-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-r9pvowy0tacjusv-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=33 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-r9pvowy0tacjusv --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 child 13597 on humgen-05-15.2 exit 1 success=
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:34:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 failure (#1, temporary) after 99 seconds
2016-12-07_01:34:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 task output (0 bytes): 
2016-12-07_01:34:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 23 done, 7 running, 193 todo
2016-12-07_01:34:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  Every node has failed -- giving up
2016-12-07_01:34:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  wait for last 7 children to finish
2016-12-07_01:34:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Have 5400 potential reusable task outputs
2016-12-07_01:34:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:34:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Have 5400 potentially reusable tasks
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Creating task to process interval: [11:104013367-119632238]
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr No reusable JobTask matched key parameters [u'124_of_200', u'd368badda25753b2dc89a57d77b90d16+14700', '11:104013367-119632238', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Ending task 1 successfully
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: blkio:0:0 0 write 12528 read -- interval 10.0000 seconds 0 write 12528 read
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr srun: Job step created
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 child 14644 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 success in 68 seconds
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 task output (0 bytes): 
2016-12-07_01:34:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-f5vx8p9vrdy4mpx-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-f5vx8p9vrdy4mpx-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=28 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-f5vx8p9vrdy4mpx --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/85c5f890482f26099a5c692132b5b511b00dbd109dd63c38293a07835845be5c/cgroup.procs
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/85c5f890482f26099a5c692132b5b511b00dbd109dd63c38293a07835845be5c/memory.stat
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5623808 rss
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/85c5f890482f26099a5c692132b5b511b00dbd109dd63c38293a07835845be5c/cpuacct.stat
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/85c5f890482f26099a5c692132b5b511b00dbd109dd63c38293a07835845be5c/cpuset.cpus
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/85c5f890482f26099a5c692132b5b511b00dbd109dd63c38293a07835845be5c/blkio.io_service_bytes
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Mounting task input collection to get interval_list
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Interval_List collection mounted at /keep/491245cb71c983164b9080ee09a6a4ed+94
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Total chunk length is 15498872
2016-12-07_01:34:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:34:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Have 1 intervals
2016-12-07_01:34:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:34:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:34:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:34:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:34:26 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 24 done, 6 running, 194 todo
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: blkio:0:0 0 write 12565 read -- interval 10.0000 seconds 0 write 12565 read
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 33906688 rss
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: cpu 0.5100 user 0.1900 sys 32 cpus -- interval 10.0002 seconds 0.4800 user 0.1700 sys
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr crunchstat: net:eth0 29995 tx 3237562 rx
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Have 5400 potential reusable task outputs
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Have 5400 potentially reusable tasks
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Creating task to process interval: [11:119632239-135006516 12:1-124594]
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr No reusable JobTask matched key parameters [u'125_of_200', u'1eac160b9cfb408e238c4373c0bf58e3+14700', '11:119632239-135006516 12:1-124594', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:34:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr Ending task 1 successfully
2016-12-07_01:34:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr srun: Job step created
2016-12-07_01:34:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:34:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 child 14959 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:34:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 success in 70 seconds
2016-12-07_01:34:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 task output (0 bytes): 
2016-12-07_01:34:30 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qdl6sjf1fvebcfg-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qdl6sjf1fvebcfg-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=29 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-qdl6sjf1fvebcfg --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/014bc0e63def774c81d7199b31d255212796133ce4ea307cca270925bfaba926/cgroup.procs
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/014bc0e63def774c81d7199b31d255212796133ce4ea307cca270925bfaba926/memory.stat
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 2265088 rss
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/014bc0e63def774c81d7199b31d255212796133ce4ea307cca270925bfaba926/cpuacct.stat
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/014bc0e63def774c81d7199b31d255212796133ce4ea307cca270925bfaba926/cpuset.cpus
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/014bc0e63def774c81d7199b31d255212796133ce4ea307cca270925bfaba926/blkio.io_service_bytes
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Mounting task input collection to get interval_list
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Interval_List collection mounted at /keep/5e8b2f2d9561837a2332e3bfe93dae5b+94
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Total chunk length is 15618872
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Have 1 intervals
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:34:37 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 25 done, 5 running, 195 todo
2016-12-07_01:34:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Have 5400 potential reusable task outputs
2016-12-07_01:34:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:34:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:34:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Have 5400 potentially reusable tasks
2016-12-07_01:34:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Creating task to process interval: [12:124595-15743466]
2016-12-07_01:34:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr No reusable JobTask matched key parameters [u'126_of_200', u'cdb0e4111be1e83b1355ae75b79fc3ad+14704', '12:124595-15743466', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:34:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr Ending task 1 successfully
2016-12-07_01:34:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 child 15286 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:34:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 success in 70 seconds
2016-12-07_01:34:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 task output (0 bytes): 
2016-12-07_01:34:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr srun: Job step created
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-4ojxt1g1nb8k256-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-4ojxt1g1nb8k256-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=30 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-4ojxt1g1nb8k256 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/6fda14ea81e5bb24346e4d6f5576ed0215a4cd1785b2983dcc7c8758a0467926/cgroup.procs
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/6fda14ea81e5bb24346e4d6f5576ed0215a4cd1785b2983dcc7c8758a0467926/memory.stat
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5697536 rss
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/6fda14ea81e5bb24346e4d6f5576ed0215a4cd1785b2983dcc7c8758a0467926/cpuacct.stat
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/6fda14ea81e5bb24346e4d6f5576ed0215a4cd1785b2983dcc7c8758a0467926/cpuset.cpus
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/6fda14ea81e5bb24346e4d6f5576ed0215a4cd1785b2983dcc7c8758a0467926/blkio.io_service_bytes
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Mounting task input collection to get interval_list
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Interval_List collection mounted at /keep/e5904dcea66543e7dc108a061ba1117f+94
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Total chunk length is 15618872
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Have 1 intervals
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:34:47 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 26 done, 4 running, 196 todo
2016-12-07_01:34:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:34:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:34:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:34:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:34:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:34:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:34:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: cpu 0.4600 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.4400 user 0.1800 sys
2016-12-07_01:34:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr crunchstat: net:eth0 21811 tx 238216 rx
2016-12-07_01:34:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Have 5400 potential reusable task outputs
2016-12-07_01:34:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:34:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:34:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Have 5400 potentially reusable tasks
2016-12-07_01:34:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Creating task to process interval: [12:15743467-31362338]
2016-12-07_01:34:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr No reusable JobTask matched key parameters [u'127_of_200', u'8df47da941cf4ccb3abf73b2b9e8c8f0+14700', '12:15743467-31362338', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:34:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr Ending task 1 successfully
2016-12-07_01:34:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 child 15891 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:34:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 success in 72 seconds
2016-12-07_01:34:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 task output (0 bytes): 
2016-12-07_01:34:52 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr srun: Job step created
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-r9pvowy0tacjusv-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-r9pvowy0tacjusv-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=33 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-r9pvowy0tacjusv --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/6047571df60f2b27f22c7e5e04019c581412739092f4b7623f97c3c469408157/cgroup.procs
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/6047571df60f2b27f22c7e5e04019c581412739092f4b7623f97c3c469408157/memory.stat
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5349376 rss
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/6047571df60f2b27f22c7e5e04019c581412739092f4b7623f97c3c469408157/cpuacct.stat
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/6047571df60f2b27f22c7e5e04019c581412739092f4b7623f97c3c469408157/cpuset.cpus
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: cpu 0.0500 user 0.0000 sys 32 cpus
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/6047571df60f2b27f22c7e5e04019c581412739092f4b7623f97c3c469408157/blkio.io_service_bytes
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Mounting task input collection to get interval_list
2016-12-07_01:34:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Interval_List collection mounted at /keep/141395361d26e08e06fe044bb6deb643+92
2016-12-07_01:34:59 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Total chunk length is 15618872
2016-12-07_01:34:59 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:34:59 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Have 1 intervals
2016-12-07_01:34:59 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:34:59 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:34:59 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:34:59 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:34:59 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 27 done, 3 running, 197 todo
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 notice: task is not in slurm queue but srun process 16264 has not exited
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: cpu 0.5000 user 0.1700 sys 32 cpus -- interval 10.0002 seconds 0.4500 user 0.1700 sys
2016-12-07_01:35:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr crunchstat: net:eth0 21853 tx 238521 rx
2016-12-07_01:35:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Have 5400 potential reusable task outputs
2016-12-07_01:35:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:35:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:35:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Have 5400 potentially reusable tasks
2016-12-07_01:35:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Creating task to process interval: [1:171807593-187426464]
2016-12-07_01:35:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr No reusable JobTask matched key parameters [u'12_of_200', u'5132bffa3f810729de8ad25529646223+14616', '1:171807593-187426464', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:35:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 stderr Ending task 1 successfully
2016-12-07_01:35:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 child 16999 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:35:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 success in 50 seconds
2016-12-07_01:35:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 task output (0 bytes): 
2016-12-07_01:35:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr srun: Job step created
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0y41phqbnenlsfs-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0y41phqbnenlsfs-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=32 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-0y41phqbnenlsfs --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/589c027fc3e80430b34bc865f649c578744189c70e3a862ba8d46e9915003843/cgroup.procs
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/589c027fc3e80430b34bc865f649c578744189c70e3a862ba8d46e9915003843/memory.stat
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5758976 rss
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/589c027fc3e80430b34bc865f649c578744189c70e3a862ba8d46e9915003843/cpuacct.stat
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/589c027fc3e80430b34bc865f649c578744189c70e3a862ba8d46e9915003843/cpuset.cpus
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/589c027fc3e80430b34bc865f649c578744189c70e3a862ba8d46e9915003843/blkio.io_service_bytes
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Mounting task input collection to get interval_list
2016-12-07_01:35:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Interval_List collection mounted at /keep/f4e4053171b2d1ad6687243c2eb1ae3e+94
2016-12-07_01:35:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Total chunk length is 15618872
2016-12-07_01:35:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:35:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Have 1 intervals
2016-12-07_01:35:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:35:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:35:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:35:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:35:12 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 28 done, 2 running, 198 todo
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Have 5400 potential reusable task outputs
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Have 5400 potentially reusable tasks
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Creating task to process interval: [12:46981211-62600082]
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr No reusable JobTask matched key parameters [u'129_of_200', u'f3efd8892b516659fc7d68a6ed27302f+14704', '12:46981211-62600082', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 49074176 rss
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: cpu 0.7100 user 0.2000 sys 32 cpus -- interval 10.0002 seconds 0.6700 user 0.2000 sys
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr crunchstat: net:eth0 38630 tx 3246496 rx
2016-12-07_01:35:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr Ending task 1 successfully
2016-12-07_01:35:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 child 16643 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:35:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 success in 71 seconds
2016-12-07_01:35:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 task output (0 bytes): 
2016-12-07_01:35:16 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:35:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:35:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr srun: Job step created
2016-12-07_01:35:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-d7p9u5ff3wkrs4o-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-d7p9u5ff3wkrs4o-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=31 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-d7p9u5ff3wkrs4o --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:35:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/17e200997b5fcf09596d94531305445b04723c1b95dff9340e59ffc02a078d2e/cgroup.procs
2016-12-07_01:35:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/17e200997b5fcf09596d94531305445b04723c1b95dff9340e59ffc02a078d2e/memory.stat
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: mem 16384 cache 0 swap 0 pgmajfault 5943296 rss
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/17e200997b5fcf09596d94531305445b04723c1b95dff9340e59ffc02a078d2e/cpuacct.stat
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/17e200997b5fcf09596d94531305445b04723c1b95dff9340e59ffc02a078d2e/cpuset.cpus
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: cpu 0.0500 user 0.0000 sys 32 cpus
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/17e200997b5fcf09596d94531305445b04723c1b95dff9340e59ffc02a078d2e/blkio.io_service_bytes
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: net:eth0 0 tx 0 rx
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Mounting task input collection to get interval_list
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Interval_List collection mounted at /keep/4433c33996669e0bdc3bf5230c5ddce5+94
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Total chunk length is 15618872
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Have 1 intervals
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:35:23 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 29 done, 1 running, 199 todo
2016-12-07_01:35:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:35:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:35:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:35:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:35:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:35:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Have 5400 potential reusable task outputs
2016-12-07_01:35:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46039040 rss
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: cpu 0.6800 user 0.1900 sys 32 cpus -- interval 10.0001 seconds 0.6300 user 0.1900 sys
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr crunchstat: net:eth0 35577 tx 3237942 rx -- interval 10.0001 seconds 35577 tx 3237942 rx
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Have 5400 potentially reusable tasks
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Creating task to process interval: [12:31362339-46981210]
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr No reusable JobTask matched key parameters [u'128_of_200', u'a8fa10c8652e6c7e83f1568bfb56cb86+14700', '12:31362339-46981210', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr Ending task 1 successfully
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 child 16264 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 success in 95 seconds
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 task output (0 bytes): 
2016-12-07_01:35:27 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:35:34 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:35:34 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 30 done, 0 running, 200 todo
2016-12-07_01:35:34 z8ta6-8i9sb-r52unvxj6avgrio 4970  collate
2016-12-07_01:35:34 z8ta6-8i9sb-r52unvxj6avgrio 4970  collated output manifest text to send to API server is 0 bytes with access tokens
2016-12-07_01:35:34 z8ta6-8i9sb-r52unvxj6avgrio 4970  job output d41d8cd98f00b204e9800998ecf8427e+0
2016-12-07_01:35:40 z8ta6-8i9sb-r52unvxj6avgrio 19831  running from /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/arvados-cli-0.1.20161017193526/bin/crunch-job with arvados-cli Gem version(s) 0.1.20161017193526, 0.1.20160503204200, 0.1.20151207150126, 0.1.20151023190001
2016-12-07_01:35:40 z8ta6-8i9sb-r52unvxj6avgrio 19831  check slurm allocation
2016-12-07_01:35:40 z8ta6-8i9sb-r52unvxj6avgrio 19831  node humgen-05-15 - 8 slots
2016-12-07_01:35:40 z8ta6-8i9sb-r52unvxj6avgrio 19831  start
2016-12-07_01:35:40 z8ta6-8i9sb-r52unvxj6avgrio 19831  clean work dirs: start
2016-12-07_01:35:40 z8ta6-8i9sb-r52unvxj6avgrio 19831  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk "(index(\\$3, \\"$CRUNCH_TMP\\") == 1){print \\$3}" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']
2016-12-07_01:35:41 z8ta6-8i9sb-r52unvxj6avgrio 19831  clean work dirs: exit 0
2016-12-07_01:35:41 z8ta6-8i9sb-r52unvxj6avgrio 19831  Install docker image 9289c4f1fce1d1dedf9cc7876c33f936+671
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  docker image hash is 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  load docker image: start
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  stderr starting: ['srun','--nodelist=humgen-05-15','/bin/bash','-o','pipefail','-ec',' if /usr/bin/docker images -q --no-trunc --all | grep -xF 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 >/dev/null; then     exit 0 fi declare -a exit_codes=("${PIPESTATUS[@]}") if [ 0 != "${exit_codes[0]}" ]; then    exit "${exit_codes[0]}"  # `docker images` failed elif [ 1 != "${exit_codes[1]}" ]; then    exit "${exit_codes[1]}"  # `grep` encountered an error else    # Everything worked fine, but grep didn\'t find the image on this host.    arv-get 9289c4f1fce1d1dedf9cc7876c33f936\\+671\\/571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3\\.tar | /usr/bin/docker load fi ']
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  load docker image: exit 0
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  check --memory-swap feature: start
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  stderr starting: ['srun','--nodes=1','/usr/bin/docker','run','--help']
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  check --memory-swap feature: exit 0
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  check whether default user is UID 0: start
2016-12-07_01:35:43 z8ta6-8i9sb-r52unvxj6avgrio 19831  stderr starting: ['srun','--nodes=1','/bin/sh','-ec','/usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21  571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 id --user']
2016-12-07_01:35:44 z8ta6-8i9sb-r52unvxj6avgrio 19831  check whether default user is UID 0: exit 0
2016-12-07_01:35:44 z8ta6-8i9sb-r52unvxj6avgrio 19831  Container will run with default user
2016-12-07_01:35:44 z8ta6-8i9sb-r52unvxj6avgrio 19831  Looking for version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 from repository jr17/hgi
2016-12-07_01:35:44 z8ta6-8i9sb-r52unvxj6avgrio 19831  Using local repository '/var/lib/arvados/internal.git'
2016-12-07_01:35:44 z8ta6-8i9sb-r52unvxj6avgrio 19831  Version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 is commit 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:35:44 z8ta6-8i9sb-r52unvxj6avgrio 19831  run install script on all workers: start
2016-12-07_01:35:44 z8ta6-8i9sb-r52unvxj6avgrio 19831  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio','sh','-c','mkdir -p /data/crunch-tmp/crunch-job/opt && cd /data/crunch-tmp/crunch-job && perl -']
2016-12-07_01:35:44 z8ta6-8i9sb-r52unvxj6avgrio 19831  stderr Extracting archive: 4b8cf8e1274f8d418cb494d3ea978152
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  run install script on all workers: exit 0
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  script gatk-combinegvcfs.py
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  script_version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  script_parameters {"interval_lists_collection":"4a72f79fd2ed51ac692bc326fb168f37+10043","reference_collection":"1b447d1aa006cde57df03042736750b9+7894","inputs_collection":"b0dd82bb014b6ab8e64968ebdc07aad9+2566847"}
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  runtime_constraints {"keep_cache_mb_per_task":19500,"max_tasks_per_node":8,"min_nodes":1,"docker_image":"mercury/gatk-3.6"}
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  start level 0 with 1 slots
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 0 done, 0 running, 1 todo
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 job_task z8ta6-ot0gb-4fxfcuiqiyhc1xm
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 child 20078 started on humgen-05-15.1
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.0.20078','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (1 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-4fxfcuiqiyhc1xm-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-4fxfcuiqiyhc1xm-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-4fxfcuiqiyhc1xm-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=0 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=0 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-4fxfcuiqiyhc1xm --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 0 done, 1 running, 0 todo
2016-12-07_01:35:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-4fxfcuiqiyhc1xm-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-4fxfcuiqiyhc1xm-0.cid --sig-proxy --memory=250853295k --memory-swap=284407723k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=0 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=0 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-4fxfcuiqiyhc1xm --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:35:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/aafb7d965affde37ce4ff36e5b3c687bac2839f26d28aa575b199f944415ad72/cgroup.procs
2016-12-07_01:35:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/aafb7d965affde37ce4ff36e5b3c687bac2839f26d28aa575b199f944415ad72/memory.stat
2016-12-07_01:35:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4579328 rss
2016-12-07_01:35:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/aafb7d965affde37ce4ff36e5b3c687bac2839f26d28aa575b199f944415ad72/cpuacct.stat
2016-12-07_01:35:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/aafb7d965affde37ce4ff36e5b3c687bac2839f26d28aa575b199f944415ad72/cpuset.cpus
2016-12-07_01:35:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:35:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/aafb7d965affde37ce4ff36e5b3c687bac2839f26d28aa575b199f944415ad72/blkio.io_service_bytes
2016-12-07_01:35:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:35:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Processing last stream
2016-12-07_01:35:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Finalising stream .
2016-12-07_01:35:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 100_of_200
2016-12-07_01:35:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 100_of_200
2016-12-07_01:35:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 101_of_200
2016-12-07_01:35:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 101_of_200
2016-12-07_01:35:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 102_of_200
2016-12-07_01:35:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:35:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:35:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:35:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:35:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:35:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 151453696 rss
2016-12-07_01:35:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 3.6200 user 0.3200 sys 32 cpus -- interval 10.0001 seconds 3.6000 user 0.3200 sys
2016-12-07_01:35:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 89205 tx 3351140 rx
2016-12-07_01:35:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 102_of_200
2016-12-07_01:35:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 103_of_200
2016-12-07_01:35:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 103_of_200
2016-12-07_01:35:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 104_of_200
2016-12-07_01:35:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 104_of_200
2016-12-07_01:35:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 105_of_200
2016-12-07_01:35:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 105_of_200
2016-12-07_01:35:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 106_of_200
2016-12-07_01:35:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 106_of_200
2016-12-07_01:35:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 107_of_200
2016-12-07_01:35:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 107_of_200
2016-12-07_01:35:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 108_of_200
2016-12-07_01:35:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 108_of_200
2016-12-07_01:35:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 109_of_200
2016-12-07_01:35:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 109_of_200
2016-12-07_01:35:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 10_of_200
2016-12-07_01:35:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 10_of_200
2016-12-07_01:36:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 110_of_200
2016-12-07_01:36:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 110_of_200
2016-12-07_01:36:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 111_of_200
2016-12-07_01:36:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 111_of_200
2016-12-07_01:36:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 112_of_200
2016-12-07_01:36:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 112_of_200
2016-12-07_01:36:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 113_of_200
2016-12-07_01:36:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 113_of_200
2016-12-07_01:36:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 114_of_200
2016-12-07_01:36:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 114_of_200
2016-12-07_01:36:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 115_of_200
2016-12-07_01:36:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 115_of_200
2016-12-07_01:36:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 116_of_200
2016-12-07_01:36:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 116_of_200
2016-12-07_01:36:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 117_of_200
2016-12-07_01:36:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 117_of_200
2016-12-07_01:36:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 118_of_200
2016-12-07_01:36:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 118_of_200
2016-12-07_01:36:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 119_of_200
2016-12-07_01:36:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 119_of_200
2016-12-07_01:36:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 11_of_200
2016-12-07_01:36:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 11_of_200
2016-12-07_01:36:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 120_of_200
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 120_of_200
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 121_of_200
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 121_of_200
2016-12-07_01:36:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 122_of_200
2016-12-07_01:36:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 175489024 rss
2016-12-07_01:36:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 6.5400 user 0.4900 sys 32 cpus -- interval 10.0000 seconds 2.9200 user 0.1700 sys
2016-12-07_01:36:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 611750 tx 4236948 rx -- interval 10.0000 seconds 522545 tx 885808 rx
2016-12-07_01:36:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 122_of_200
2016-12-07_01:36:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 123_of_200
2016-12-07_01:36:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 123_of_200
2016-12-07_01:36:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 124_of_200
2016-12-07_01:36:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 124_of_200
2016-12-07_01:36:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 125_of_200
2016-12-07_01:36:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 125_of_200
2016-12-07_01:36:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 126_of_200
2016-12-07_01:36:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 126_of_200
2016-12-07_01:36:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 127_of_200
2016-12-07_01:36:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 127_of_200
2016-12-07_01:36:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 128_of_200
2016-12-07_01:36:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 128_of_200
2016-12-07_01:36:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 129_of_200
2016-12-07_01:36:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 129_of_200
2016-12-07_01:36:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 12_of_200
2016-12-07_01:36:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 12_of_200
2016-12-07_01:36:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 130_of_200
2016-12-07_01:36:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 130_of_200
2016-12-07_01:36:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 131_of_200
2016-12-07_01:36:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 131_of_200
2016-12-07_01:36:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 132_of_200
2016-12-07_01:36:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 132_of_200
2016-12-07_01:36:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 133_of_200
2016-12-07_01:36:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 133_of_200
2016-12-07_01:36:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 134_of_200
2016-12-07_01:36:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 134_of_200
2016-12-07_01:36:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 135_of_200
2016-12-07_01:36:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 135_of_200
2016-12-07_01:36:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 136_of_200
2016-12-07_01:36:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 136_of_200
2016-12-07_01:36:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 137_of_200
2016-12-07_01:36:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 137_of_200
2016-12-07_01:36:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 138_of_200
2016-12-07_01:36:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 138_of_200
2016-12-07_01:36:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 139_of_200
2016-12-07_01:36:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 139_of_200
2016-12-07_01:36:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 13_of_200
2016-12-07_01:36:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 13_of_200
2016-12-07_01:36:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 140_of_200
2016-12-07_01:36:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 140_of_200
2016-12-07_01:36:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 141_of_200
2016-12-07_01:36:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 141_of_200
2016-12-07_01:36:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 142_of_200
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 142_of_200
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 143_of_200
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 143_of_200
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 144_of_200
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 144_of_200
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 179347456 rss
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 9.9700 user 0.6800 sys 32 cpus -- interval 10.0000 seconds 3.4300 user 0.1900 sys
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 1184657 tx 5225606 rx -- interval 10.0000 seconds 572907 tx 988658 rx
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 145_of_200
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 145_of_200
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 146_of_200
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 146_of_200
2016-12-07_01:36:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 147_of_200
2016-12-07_01:36:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 147_of_200
2016-12-07_01:36:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 148_of_200
2016-12-07_01:36:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 148_of_200
2016-12-07_01:36:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 149_of_200
2016-12-07_01:36:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 149_of_200
2016-12-07_01:36:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 14_of_200
2016-12-07_01:36:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 14_of_200
2016-12-07_01:36:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 150_of_200
2016-12-07_01:36:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 150_of_200
2016-12-07_01:36:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 151_of_200
2016-12-07_01:36:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 151_of_200
2016-12-07_01:36:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 152_of_200
2016-12-07_01:36:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 152_of_200
2016-12-07_01:36:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 153_of_200
2016-12-07_01:36:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 153_of_200
2016-12-07_01:36:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 154_of_200
2016-12-07_01:36:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 154_of_200
2016-12-07_01:36:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 155_of_200
2016-12-07_01:36:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 155_of_200
2016-12-07_01:36:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 156_of_200
2016-12-07_01:36:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 156_of_200
2016-12-07_01:36:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 157_of_200
2016-12-07_01:36:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 157_of_200
2016-12-07_01:36:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 158_of_200
2016-12-07_01:36:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 158_of_200
2016-12-07_01:36:22 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 159_of_200
2016-12-07_01:36:22 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 159_of_200
2016-12-07_01:36:22 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 15_of_200
2016-12-07_01:36:22 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 15_of_200
2016-12-07_01:36:22 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 160_of_200
2016-12-07_01:36:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 160_of_200
2016-12-07_01:36:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 161_of_200
2016-12-07_01:36:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 161_of_200
2016-12-07_01:36:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 162_of_200
2016-12-07_01:36:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 162_of_200
2016-12-07_01:36:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 163_of_200
2016-12-07_01:36:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 163_of_200
2016-12-07_01:36:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 164_of_200
2016-12-07_01:36:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 164_of_200
2016-12-07_01:36:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 165_of_200
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 165_of_200
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 166_of_200
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 166_of_200
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 167_of_200
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 167_of_200
2016-12-07_01:36:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 205750272 rss
2016-12-07_01:36:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 13.3400 user 0.8400 sys 32 cpus -- interval 10.0000 seconds 3.3700 user 0.1600 sys
2016-12-07_01:36:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 1779566 tx 6237813 rx -- interval 10.0000 seconds 594909 tx 1012207 rx
2016-12-07_01:36:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 168_of_200
2016-12-07_01:36:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 168_of_200
2016-12-07_01:36:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 169_of_200
2016-12-07_01:36:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 169_of_200
2016-12-07_01:36:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 16_of_200
2016-12-07_01:36:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 16_of_200
2016-12-07_01:36:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 170_of_200
2016-12-07_01:36:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 170_of_200
2016-12-07_01:36:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 171_of_200
2016-12-07_01:36:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 171_of_200
2016-12-07_01:36:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 172_of_200
2016-12-07_01:36:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 172_of_200
2016-12-07_01:36:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 173_of_200
2016-12-07_01:36:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 173_of_200
2016-12-07_01:36:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 174_of_200
2016-12-07_01:36:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 174_of_200
2016-12-07_01:36:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 175_of_200
2016-12-07_01:36:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 175_of_200
2016-12-07_01:36:30 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 176_of_200
2016-12-07_01:36:30 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 176_of_200
2016-12-07_01:36:30 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 177_of_200
2016-12-07_01:36:30 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 177_of_200
2016-12-07_01:36:30 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 178_of_200
2016-12-07_01:36:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 178_of_200
2016-12-07_01:36:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 179_of_200
2016-12-07_01:36:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 179_of_200
2016-12-07_01:36:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 17_of_200
2016-12-07_01:36:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 17_of_200
2016-12-07_01:36:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 180_of_200
2016-12-07_01:36:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 180_of_200
2016-12-07_01:36:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 181_of_200
2016-12-07_01:36:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 181_of_200
2016-12-07_01:36:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 182_of_200
2016-12-07_01:36:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 182_of_200
2016-12-07_01:36:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 183_of_200
2016-12-07_01:36:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 183_of_200
2016-12-07_01:36:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 184_of_200
2016-12-07_01:36:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 184_of_200
2016-12-07_01:36:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 185_of_200
2016-12-07_01:36:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 185_of_200
2016-12-07_01:36:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 186_of_200
2016-12-07_01:36:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 186_of_200
2016-12-07_01:36:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 187_of_200
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 187_of_200
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 188_of_200
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 188_of_200
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 189_of_200
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 189_of_200
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 235741184 rss
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 16.4800 user 1.0000 sys 32 cpus -- interval 10.0000 seconds 3.1400 user 0.1600 sys
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 2348497 tx 7199325 rx -- interval 10.0000 seconds 568931 tx 961512 rx
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 18_of_200
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 18_of_200
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 190_of_200
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 190_of_200
2016-12-07_01:36:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 191_of_200
2016-12-07_01:36:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 191_of_200
2016-12-07_01:36:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 192_of_200
2016-12-07_01:36:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 192_of_200
2016-12-07_01:36:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 193_of_200
2016-12-07_01:36:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 193_of_200
2016-12-07_01:36:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 194_of_200
2016-12-07_01:36:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 194_of_200
2016-12-07_01:36:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 195_of_200
2016-12-07_01:36:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 195_of_200
2016-12-07_01:36:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 196_of_200
2016-12-07_01:36:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 196_of_200
2016-12-07_01:36:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 197_of_200
2016-12-07_01:36:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 197_of_200
2016-12-07_01:36:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 198_of_200
2016-12-07_01:36:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 198_of_200
2016-12-07_01:36:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 199_of_200
2016-12-07_01:36:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 199_of_200
2016-12-07_01:36:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 19_of_200
2016-12-07_01:36:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 19_of_200
2016-12-07_01:36:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 1_of_200
2016-12-07_01:36:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 1_of_200
2016-12-07_01:36:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 200_of_200
2016-12-07_01:36:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 200_of_200
2016-12-07_01:36:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 20_of_200
2016-12-07_01:36:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 20_of_200
2016-12-07_01:36:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 21_of_200
2016-12-07_01:36:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 21_of_200
2016-12-07_01:36:43 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 22_of_200
2016-12-07_01:36:43 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 22_of_200
2016-12-07_01:36:43 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 23_of_200
2016-12-07_01:36:43 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 23_of_200
2016-12-07_01:36:43 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 24_of_200
2016-12-07_01:36:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 24_of_200
2016-12-07_01:36:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 25_of_200
2016-12-07_01:36:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 25_of_200
2016-12-07_01:36:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 26_of_200
2016-12-07_01:36:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 26_of_200
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 27_of_200
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 27_of_200
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 28_of_200
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 28_of_200
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 29_of_200
2016-12-07_01:36:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 263593984 rss
2016-12-07_01:36:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 19.6600 user 1.1500 sys 32 cpus -- interval 10.0000 seconds 3.1800 user 0.1500 sys
2016-12-07_01:36:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 2895699 tx 8137889 rx -- interval 10.0000 seconds 547202 tx 938564 rx
2016-12-07_01:36:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 29_of_200
2016-12-07_01:36:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 2_of_200
2016-12-07_01:36:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 2_of_200
2016-12-07_01:36:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 30_of_200
2016-12-07_01:36:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 30_of_200
2016-12-07_01:36:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 31_of_200
2016-12-07_01:36:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 31_of_200
2016-12-07_01:36:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 32_of_200
2016-12-07_01:36:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 32_of_200
2016-12-07_01:36:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 33_of_200
2016-12-07_01:36:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 33_of_200
2016-12-07_01:36:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 34_of_200
2016-12-07_01:36:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 34_of_200
2016-12-07_01:36:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 35_of_200
2016-12-07_01:36:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 35_of_200
2016-12-07_01:36:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 36_of_200
2016-12-07_01:36:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 36_of_200
2016-12-07_01:36:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 37_of_200
2016-12-07_01:36:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 37_of_200
2016-12-07_01:36:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 38_of_200
2016-12-07_01:36:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 38_of_200
2016-12-07_01:36:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 39_of_200
2016-12-07_01:36:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 39_of_200
2016-12-07_01:36:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 3_of_200
2016-12-07_01:36:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 3_of_200
2016-12-07_01:36:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 40_of_200
2016-12-07_01:36:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 40_of_200
2016-12-07_01:36:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 41_of_200
2016-12-07_01:36:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 41_of_200
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 42_of_200
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 42_of_200
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 43_of_200
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 43_of_200
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:36:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 44_of_200
2016-12-07_01:36:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 282341376 rss
2016-12-07_01:36:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 21.8800 user 1.2800 sys 32 cpus -- interval 10.0000 seconds 2.2200 user 0.1300 sys
2016-12-07_01:36:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 3298032 tx 8821899 rx -- interval 10.0000 seconds 402333 tx 684010 rx
2016-12-07_01:36:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 44_of_200
2016-12-07_01:36:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 45_of_200
2016-12-07_01:36:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 45_of_200
2016-12-07_01:36:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 46_of_200
2016-12-07_01:36:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 46_of_200
2016-12-07_01:36:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 47_of_200
2016-12-07_01:36:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 47_of_200
2016-12-07_01:36:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 48_of_200
2016-12-07_01:36:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 48_of_200
2016-12-07_01:36:57 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 49_of_200
2016-12-07_01:36:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 49_of_200
2016-12-07_01:36:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 4_of_200
2016-12-07_01:36:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 4_of_200
2016-12-07_01:36:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 50_of_200
2016-12-07_01:36:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 50_of_200
2016-12-07_01:36:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 51_of_200
2016-12-07_01:36:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 51_of_200
2016-12-07_01:36:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 52_of_200
2016-12-07_01:36:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 52_of_200
2016-12-07_01:37:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 53_of_200
2016-12-07_01:37:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 53_of_200
2016-12-07_01:37:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 54_of_200
2016-12-07_01:37:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 54_of_200
2016-12-07_01:37:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 55_of_200
2016-12-07_01:37:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 55_of_200
2016-12-07_01:37:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 56_of_200
2016-12-07_01:37:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 56_of_200
2016-12-07_01:37:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 57_of_200
2016-12-07_01:37:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 57_of_200
2016-12-07_01:37:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 58_of_200
2016-12-07_01:37:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 58_of_200
2016-12-07_01:37:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 59_of_200
2016-12-07_01:37:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 59_of_200
2016-12-07_01:37:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 5_of_200
2016-12-07_01:37:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 5_of_200
2016-12-07_01:37:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 60_of_200
2016-12-07_01:37:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 60_of_200
2016-12-07_01:37:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 61_of_200
2016-12-07_01:37:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 61_of_200
2016-12-07_01:37:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 62_of_200
2016-12-07_01:37:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 62_of_200
2016-12-07_01:37:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 63_of_200
2016-12-07_01:37:04 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 63_of_200
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 64_of_200
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 64_of_200
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 65_of_200
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 65_of_200
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 66_of_200
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:37:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 66_of_200
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 314134528 rss
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 25.1700 user 1.4400 sys 32 cpus -- interval 10.0000 seconds 3.2900 user 0.1600 sys
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 3888512 tx 9818566 rx -- interval 10.0000 seconds 590480 tx 996667 rx
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 67_of_200
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 67_of_200
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 68_of_200
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 68_of_200
2016-12-07_01:37:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 69_of_200
2016-12-07_01:37:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 69_of_200
2016-12-07_01:37:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 6_of_200
2016-12-07_01:37:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 6_of_200
2016-12-07_01:37:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 70_of_200
2016-12-07_01:37:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 70_of_200
2016-12-07_01:37:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 71_of_200
2016-12-07_01:37:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 71_of_200
2016-12-07_01:37:08 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 72_of_200
2016-12-07_01:37:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 72_of_200
2016-12-07_01:37:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 73_of_200
2016-12-07_01:37:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 73_of_200
2016-12-07_01:37:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 74_of_200
2016-12-07_01:37:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 74_of_200
2016-12-07_01:37:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 75_of_200
2016-12-07_01:37:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 75_of_200
2016-12-07_01:37:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 76_of_200
2016-12-07_01:37:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 76_of_200
2016-12-07_01:37:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 77_of_200
2016-12-07_01:37:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 77_of_200
2016-12-07_01:37:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 78_of_200
2016-12-07_01:37:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 78_of_200
2016-12-07_01:37:11 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 79_of_200
2016-12-07_01:37:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 79_of_200
2016-12-07_01:37:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 7_of_200
2016-12-07_01:37:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 7_of_200
2016-12-07_01:37:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 80_of_200
2016-12-07_01:37:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 80_of_200
2016-12-07_01:37:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 81_of_200
2016-12-07_01:37:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 81_of_200
2016-12-07_01:37:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 82_of_200
2016-12-07_01:37:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 82_of_200
2016-12-07_01:37:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 83_of_200
2016-12-07_01:37:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 83_of_200
2016-12-07_01:37:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 84_of_200
2016-12-07_01:37:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 84_of_200
2016-12-07_01:37:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 85_of_200
2016-12-07_01:37:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 85_of_200
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 86_of_200
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 86_of_200
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 87_of_200
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 87_of_200
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:37:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 88_of_200
2016-12-07_01:37:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 308654080 rss
2016-12-07_01:37:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: cpu 28.7500 user 1.6700 sys 32 cpus -- interval 9.9999 seconds 3.5800 user 0.2300 sys
2016-12-07_01:37:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr crunchstat: net:eth0 4451446 tx 10760551 rx -- interval 9.9999 seconds 562934 tx 941985 rx
2016-12-07_01:37:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 88_of_200
2016-12-07_01:37:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 89_of_200
2016-12-07_01:37:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 89_of_200
2016-12-07_01:37:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 8_of_200
2016-12-07_01:37:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 8_of_200
2016-12-07_01:37:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 90_of_200
2016-12-07_01:37:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 90_of_200
2016-12-07_01:37:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 91_of_200
2016-12-07_01:37:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 91_of_200
2016-12-07_01:37:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 92_of_200
2016-12-07_01:37:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 92_of_200
2016-12-07_01:37:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 93_of_200
2016-12-07_01:37:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 93_of_200
2016-12-07_01:37:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 94_of_200
2016-12-07_01:37:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 94_of_200
2016-12-07_01:37:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 95_of_200
2016-12-07_01:37:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 95_of_200
2016-12-07_01:37:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 96_of_200
2016-12-07_01:37:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 96_of_200
2016-12-07_01:37:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 97_of_200
2016-12-07_01:37:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 97_of_200
2016-12-07_01:37:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 98_of_200
2016-12-07_01:37:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 98_of_200
2016-12-07_01:37:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 99_of_200
2016-12-07_01:37:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 99_of_200
2016-12-07_01:37:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Have 42 gVCFs in group 9_of_200
2016-12-07_01:37:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Creating task to process 9_of_200
2016-12-07_01:37:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 stderr Ending task 0 successfully
2016-12-07_01:37:22 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 child 20078 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:37:22 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 success in 97 seconds
2016-12-07_01:37:22 z8ta6-8i9sb-r52unvxj6avgrio 19831 0 task output (0 bytes): 
2016-12-07_01:37:22 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 200 new tasks from API server.
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 1 done, 0 running, 200 todo
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831  wait for last 0 children to finish
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831  start level 1 with 8 slots
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 1 done, 0 running, 200 todo
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 job_task z8ta6-ot0gb-sd53g6o7460omsd
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 child 23256 started on humgen-05-15.1
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 job_task z8ta6-ot0gb-nqzldhjp7mb60of
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 child 23273 started on humgen-05-15.2
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 job_task z8ta6-ot0gb-8fikxrmqw5myg22
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 child 23277 started on humgen-05-15.3
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 job_task z8ta6-ot0gb-xkjy812oqcaoov0
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 child 23301 started on humgen-05-15.4
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 job_task z8ta6-ot0gb-k7wgi5wi74c39br
2016-12-07_01:37:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 child 23319 started on humgen-05-15.5
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 job_task z8ta6-ot0gb-twu3y9j25n1gnzv
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 child 23337 started on humgen-05-15.6
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 job_task z8ta6-ot0gb-bdz4831ruh5ujyf
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 child 23341 started on humgen-05-15.7
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 job_task z8ta6-ot0gb-tn99pprx4lci5s4
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 child 23358 started on humgen-05-15.8
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.1.23256','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-sd53g6o7460omsd-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-sd53g6o7460omsd-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-sd53g6o7460omsd-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=1 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-sd53g6o7460omsd --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-sd53g6o7460omsd-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-sd53g6o7460omsd-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=1 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-sd53g6o7460omsd --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/668eeb75e0311176beab1abc2d4f7b0c06fe2d81ba4edb5d28cb169e5fe65f6b/cgroup.procs
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/668eeb75e0311176beab1abc2d4f7b0c06fe2d81ba4edb5d28cb169e5fe65f6b/memory.stat
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4333568 rss
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/668eeb75e0311176beab1abc2d4f7b0c06fe2d81ba4edb5d28cb169e5fe65f6b/cpuacct.stat
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/668eeb75e0311176beab1abc2d4f7b0c06fe2d81ba4edb5d28cb169e5fe65f6b/cpuset.cpus
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/668eeb75e0311176beab1abc2d4f7b0c06fe2d81ba4edb5d28cb169e5fe65f6b/blkio.io_service_bytes
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.2.23273','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-nqzldhjp7mb60of-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-nqzldhjp7mb60of-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-nqzldhjp7mb60of-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=2 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-nqzldhjp7mb60of --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.3.23277','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8fikxrmqw5myg22-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8fikxrmqw5myg22-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8fikxrmqw5myg22-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=3 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-8fikxrmqw5myg22 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.4.23301','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xkjy812oqcaoov0-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xkjy812oqcaoov0-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xkjy812oqcaoov0-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=4 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-xkjy812oqcaoov0 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.5.23319','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k7wgi5wi74c39br-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-k7wgi5wi74c39br-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k7wgi5wi74c39br-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=5 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-k7wgi5wi74c39br --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.6.23337','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-twu3y9j25n1gnzv-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-twu3y9j25n1gnzv-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-twu3y9j25n1gnzv-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=6 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-twu3y9j25n1gnzv --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.7.23341','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-bdz4831ruh5ujyf-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-bdz4831ruh5ujyf-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-bdz4831ruh5ujyf-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=7 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-bdz4831ruh5ujyf --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.8.23358','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-tn99pprx4lci5s4-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-tn99pprx4lci5s4-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-tn99pprx4lci5s4-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=8 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-tn99pprx4lci5s4 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 1 done, 8 running, 192 todo
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Mounting task input collection to get interval_list
2016-12-07_01:37:33 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Interval_List collection mounted at /keep/812316f6c95d8641412912523842bbfc+94
2016-12-07_01:37:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Total chunk length is 15618872
2016-12-07_01:37:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:37:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Have 1 intervals
2016-12-07_01:37:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:37:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:37:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:37:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:37:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Have 5400 potential reusable task outputs
2016-12-07_01:37:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:37:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:37:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Have 5400 potentially reusable tasks
2016-12-07_01:37:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Creating task to process interval: [9:6148617-21767488]
2016-12-07_01:37:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr No reusable JobTask matched key parameters [u'100_of_200', u'a241297995deeab501f9bd99dc533315+14700', '9:6148617-21767488', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:37:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 stderr Ending task 1 successfully
2016-12-07_01:37:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 child 23256 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:37:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 success in 10 seconds
2016-12-07_01:37:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 1 task output (0 bytes): 
2016-12-07_01:37:42 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:37:49 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:37:49 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 2 done, 7 running, 193 todo
2016-12-07_01:37:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 job_task z8ta6-ot0gb-vcrhudqfkkhqbf9
2016-12-07_01:37:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 child 23828 started on humgen-05-15.1
2016-12-07_01:37:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.9.23828','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-vcrhudqfkkhqbf9-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-vcrhudqfkkhqbf9-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-vcrhudqfkkhqbf9-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=9 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-vcrhudqfkkhqbf9 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:37:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-nqzldhjp7mb60of-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-nqzldhjp7mb60of-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=2 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-nqzldhjp7mb60of --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7dbe63a953fb9a0fe8c74058ea46474b87b8b9ec308dd83fb93f45c040433963/cgroup.procs
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/7dbe63a953fb9a0fe8c74058ea46474b87b8b9ec308dd83fb93f45c040433963/memory.stat
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4788224 rss
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7dbe63a953fb9a0fe8c74058ea46474b87b8b9ec308dd83fb93f45c040433963/cpuacct.stat
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/7dbe63a953fb9a0fe8c74058ea46474b87b8b9ec308dd83fb93f45c040433963/cpuset.cpus
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/7dbe63a953fb9a0fe8c74058ea46474b87b8b9ec308dd83fb93f45c040433963/blkio.io_service_bytes
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Mounting task input collection to get interval_list
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Interval_List collection mounted at /keep/6af141a235ccb024f6a32fc12ee98dcf+94
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Total chunk length is 15618872
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Have 1 intervals
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:37:50 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 2 done, 8 running, 192 todo
2016-12-07_01:37:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:37:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:37:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:37:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:37:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:37:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24092672 rss
2016-12-07_01:37:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: cpu 0.4800 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4500 user 0.1600 sys
2016-12-07_01:37:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr crunchstat: net:eth0 21943 tx 238413 rx
2016-12-07_01:37:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Have 5400 potential reusable task outputs
2016-12-07_01:37:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:37:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:37:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Have 5400 potentially reusable tasks
2016-12-07_01:37:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Creating task to process interval: [9:21767489-37386360]
2016-12-07_01:37:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr No reusable JobTask matched key parameters [u'101_of_200', u'1b848222e216cbae87e6932d5820a285+14700', '9:21767489-37386360', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:37:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 stderr Ending task 1 successfully
2016-12-07_01:37:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr srun: Job step created
2016-12-07_01:37:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 child 23273 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:37:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 success in 24 seconds
2016-12-07_01:37:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 2 task output (0 bytes): 
2016-12-07_01:37:56 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:38:02 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 3 done, 7 running, 193 todo
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 job_task z8ta6-ot0gb-y0ztr2oa1q1wm0q
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 child 24214 started on humgen-05-15.2
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.10.24214','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-y0ztr2oa1q1wm0q-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-y0ztr2oa1q1wm0q-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-y0ztr2oa1q1wm0q-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=10 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-y0ztr2oa1q1wm0q --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8fikxrmqw5myg22-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8fikxrmqw5myg22-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=3 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-8fikxrmqw5myg22 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7bfe8d4226a4212b9569cf0cc15f183b0324a1c9250964131b0b8fa2424f232a/cgroup.procs
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/7bfe8d4226a4212b9569cf0cc15f183b0324a1c9250964131b0b8fa2424f232a/memory.stat
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 8175616 rss
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7bfe8d4226a4212b9569cf0cc15f183b0324a1c9250964131b0b8fa2424f232a/cpuacct.stat
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/7bfe8d4226a4212b9569cf0cc15f183b0324a1c9250964131b0b8fa2424f232a/cpuset.cpus
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/7bfe8d4226a4212b9569cf0cc15f183b0324a1c9250964131b0b8fa2424f232a/blkio.io_service_bytes
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Mounting task input collection to get interval_list
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Interval_List collection mounted at /keep/2e7cf8719961c4cc5b2ffdd92ff99dd0+94
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Total chunk length is 15618872
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Have 1 intervals
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:38:03 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 3 done, 8 running, 192 todo
2016-12-07_01:38:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Have 5400 potential reusable task outputs
2016-12-07_01:38:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:38:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:38:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Have 5400 potentially reusable tasks
2016-12-07_01:38:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Creating task to process interval: [9:37386361-53005232]
2016-12-07_01:38:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr No reusable JobTask matched key parameters [u'102_of_200', u'0b01f4aa638023c7844e14aa840fe4cc+14532', '9:37386361-53005232', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:38:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 stderr Ending task 1 successfully
2016-12-07_01:38:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 child 23277 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:38:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 success in 33 seconds
2016-12-07_01:38:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 3 task output (0 bytes): 
2016-12-07_01:38:06 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:38:12 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:38:12 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 4 done, 7 running, 193 todo
2016-12-07_01:38:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 job_task z8ta6-ot0gb-ixn1co1tcqpoeuw
2016-12-07_01:38:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 child 24557 started on humgen-05-15.3
2016-12-07_01:38:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:38:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:38:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.11.24557','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ixn1co1tcqpoeuw-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ixn1co1tcqpoeuw-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ixn1co1tcqpoeuw-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=11 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ixn1co1tcqpoeuw --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:38:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr srun: Job step created
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-k7wgi5wi74c39br-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k7wgi5wi74c39br-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=5 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-k7wgi5wi74c39br --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/e44a707a43d4a0384b61be83b5f91c112a1a8192f061136c670722e7ae92f240/cgroup.procs
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/e44a707a43d4a0384b61be83b5f91c112a1a8192f061136c670722e7ae92f240/memory.stat
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: mem 94208 cache 0 swap 0 pgmajfault 5844992 rss
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/e44a707a43d4a0384b61be83b5f91c112a1a8192f061136c670722e7ae92f240/cpuacct.stat
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/e44a707a43d4a0384b61be83b5f91c112a1a8192f061136c670722e7ae92f240/cpuset.cpus
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/e44a707a43d4a0384b61be83b5f91c112a1a8192f061136c670722e7ae92f240/blkio.io_service_bytes
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Mounting task input collection to get interval_list
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Interval_List collection mounted at /keep/61144aa3079bc748eb474251bb12d2af+94
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Total chunk length is 15618872
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Have 1 intervals
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:38:13 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 4 done, 8 running, 192 todo
2016-12-07_01:38:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Have 5400 potential reusable task outputs
2016-12-07_01:38:15 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Have 5400 potentially reusable tasks
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Creating task to process interval: [9:68624105-84242976]
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr No reusable JobTask matched key parameters [u'104_of_200', u'0e7a8a6d12ed4d9a7f7a47c972f98c08+14700', '9:68624105-84242976', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr Ending task 1 successfully
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr srun: Job step created
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 child 23319 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 success in 44 seconds
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831 5 task output (0 bytes): 
2016-12-07_01:38:16 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 5 done, 7 running, 193 todo
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 job_task z8ta6-ot0gb-8b8dww9azwpsvqq
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 child 24879 started on humgen-05-15.5
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-y0ztr2oa1q1wm0q-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-y0ztr2oa1q1wm0q-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=10 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-y0ztr2oa1q1wm0q --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/b63dfa1ae3a656488f89abc4efeb8935c0bc4b31a844a614a56935bcebf66c34/cgroup.procs
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/b63dfa1ae3a656488f89abc4efeb8935c0bc4b31a844a614a56935bcebf66c34/memory.stat
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5124096 rss
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/b63dfa1ae3a656488f89abc4efeb8935c0bc4b31a844a614a56935bcebf66c34/cpuacct.stat
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/b63dfa1ae3a656488f89abc4efeb8935c0bc4b31a844a614a56935bcebf66c34/cpuset.cpus
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/b63dfa1ae3a656488f89abc4efeb8935c0bc4b31a844a614a56935bcebf66c34/blkio.io_service_bytes
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Mounting task input collection to get interval_list
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Interval_List collection mounted at /keep/8f01b9c543342e361e6b7738dc97925d+94
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Total chunk length is 15618872
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Have 1 intervals
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:38:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:38:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:38:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.12.24879','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8b8dww9azwpsvqq-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8b8dww9azwpsvqq-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8b8dww9azwpsvqq-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=12 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-8b8dww9azwpsvqq --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:38:24 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 5 done, 8 running, 192 todo
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: cpu 0.5200 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.4800 user 0.1500 sys
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr crunchstat: net:eth0 22009 tx 238282 rx
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Have 5400 potential reusable task outputs
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Have 5400 potentially reusable tasks
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Creating task to process interval: [10:5385034-21003905]
2016-12-07_01:38:27 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr No reusable JobTask matched key parameters [u'109_of_200', u'b8ca1138c80f500be618798294067825+14700', '10:5385034-21003905', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:38:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 stderr Ending task 1 successfully
2016-12-07_01:38:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 child 24214 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:38:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 success in 25 seconds
2016-12-07_01:38:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 10 task output (0 bytes): 
2016-12-07_01:38:28 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 6 done, 7 running, 193 todo
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 job_task z8ta6-ot0gb-2w80gz76vf57ljk
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 child 25497 started on humgen-05-15.2
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.13.25497','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2w80gz76vf57ljk-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2w80gz76vf57ljk-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2w80gz76vf57ljk-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=13 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2w80gz76vf57ljk --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr srun: Job step created
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-twu3y9j25n1gnzv-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-twu3y9j25n1gnzv-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=6 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-twu3y9j25n1gnzv --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/22924cb182c3aa747e49f080a4739b0d92e760d54973f561ca7401530bc45f54/cgroup.procs
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/22924cb182c3aa747e49f080a4739b0d92e760d54973f561ca7401530bc45f54/memory.stat
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr crunchstat: mem 28672 cache 0 swap 1 pgmajfault 2764800 rss
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/22924cb182c3aa747e49f080a4739b0d92e760d54973f561ca7401530bc45f54/cpuacct.stat
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/22924cb182c3aa747e49f080a4739b0d92e760d54973f561ca7401530bc45f54/cpuset.cpus
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr crunchstat: cpu 0.0200 user 0.0100 sys 32 cpus
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/22924cb182c3aa747e49f080a4739b0d92e760d54973f561ca7401530bc45f54/blkio.io_service_bytes
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Mounting task input collection to get interval_list
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Interval_List collection mounted at /keep/4325c7dac5c39f2db578e849695a12aa+94
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Total chunk length is 15618872
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Have 1 intervals
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:38:35 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 6 done, 8 running, 192 todo
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Have 5400 potential reusable task outputs
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 notice: task is not in slurm queue but srun process 23358 has not exited
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 notice: task is not in slurm queue but srun process 23341 has not exited
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 notice: task is not in slurm queue but srun process 23301 has not exited
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Have 5400 potentially reusable tasks
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Creating task to process interval: [9:84242977-99861848]
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr No reusable JobTask matched key parameters [u'105_of_200', u'3a3bb24e84ba30a06b3d46caa3b1c040+14700', '9:84242977-99861848', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 stderr Ending task 1 successfully
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 child 23337 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 success in 65 seconds
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831 6 task output (0 bytes): 
2016-12-07_01:38:38 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 7 done, 7 running, 193 todo
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 job_task z8ta6-ot0gb-dwe3asp6mogvgwm
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 child 25848 started on humgen-05-15.6
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr srun: Job step created
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ixn1co1tcqpoeuw-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ixn1co1tcqpoeuw-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=11 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ixn1co1tcqpoeuw --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/1af8ea8251adb51ffa2950a55ba2842ebf20e27c49cebc63156d96232692b315/cgroup.procs
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/1af8ea8251adb51ffa2950a55ba2842ebf20e27c49cebc63156d96232692b315/memory.stat
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5287936 rss
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/1af8ea8251adb51ffa2950a55ba2842ebf20e27c49cebc63156d96232692b315/cpuacct.stat
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/1af8ea8251adb51ffa2950a55ba2842ebf20e27c49cebc63156d96232692b315/cpuset.cpus
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr crunchstat: cpu 0.0100 user 0.0000 sys 32 cpus
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/1af8ea8251adb51ffa2950a55ba2842ebf20e27c49cebc63156d96232692b315/blkio.io_service_bytes
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:38:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Mounting task input collection to get interval_list
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Interval_List collection mounted at /keep/47924fc036d6a931b5f0cf53440e0969+92
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Total chunk length is 15618872
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Have 1 intervals
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.14.25848','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dwe3asp6mogvgwm-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-dwe3asp6mogvgwm-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dwe3asp6mogvgwm-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=14 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-dwe3asp6mogvgwm --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:38:46 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 7 done, 8 running, 192 todo
2016-12-07_01:38:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Have 5400 potential reusable task outputs
2016-12-07_01:38:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:38:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:38:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Have 5400 potentially reusable tasks
2016-12-07_01:38:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Creating task to process interval: [1:140569849-156188720]
2016-12-07_01:38:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr No reusable JobTask matched key parameters [u'10_of_200', u'a99a25ff7de535672c0817451b92bd1a+14616', '1:140569849-156188720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:38:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 stderr Ending task 1 successfully
2016-12-07_01:38:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:38:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:38:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr srun: Job step created
2016-12-07_01:38:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 child 24557 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:38:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 success in 37 seconds
2016-12-07_01:38:49 z8ta6-8i9sb-r52unvxj6avgrio 19831 11 task output (0 bytes): 
2016-12-07_01:38:49 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:38:55 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:38:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 notice: task is not in slurm queue but srun process 23828 has not exited
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 8 done, 7 running, 193 todo
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 job_task z8ta6-ot0gb-mj9f5fdhuw02abp
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 child 26154 started on humgen-05-15.3
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.15.26154','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-mj9f5fdhuw02abp-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-mj9f5fdhuw02abp-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-mj9f5fdhuw02abp-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=15 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-mj9f5fdhuw02abp --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-bdz4831ruh5ujyf-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-bdz4831ruh5ujyf-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=7 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-bdz4831ruh5ujyf --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/b046e6b318d1b440a73021b27c4a4b34ad18861f68aeee37535652a46fa01378/cgroup.procs
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/b046e6b318d1b440a73021b27c4a4b34ad18861f68aeee37535652a46fa01378/memory.stat
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4866048 rss
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/b046e6b318d1b440a73021b27c4a4b34ad18861f68aeee37535652a46fa01378/cpuacct.stat
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/b046e6b318d1b440a73021b27c4a4b34ad18861f68aeee37535652a46fa01378/cpuset.cpus
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/b046e6b318d1b440a73021b27c4a4b34ad18861f68aeee37535652a46fa01378/blkio.io_service_bytes
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Mounting task input collection to get interval_list
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Interval_List collection mounted at /keep/34f0119dab554faf0acfc04f3cae7c1d+94
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Total chunk length is 15618872
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Have 1 intervals
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:38:56 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 8 done, 8 running, 192 todo
2016-12-07_01:38:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Have 5400 potential reusable task outputs
2016-12-07_01:38:58 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Have 5400 potentially reusable tasks
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Creating task to process interval: [9:99861849-115480720]
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr No reusable JobTask matched key parameters [u'106_of_200', u'dc8fa207ac10a02e00e1ba8ab1687ad9+14700', '9:99861849-115480720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr Ending task 1 successfully
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 child 23341 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 success in 86 seconds
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 7 task output (0 bytes): 
2016-12-07_01:38:59 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 9 done, 7 running, 193 todo
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 job_task z8ta6-ot0gb-48tg8mv5801snwy
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 child 26464 started on humgen-05-15.7
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr srun: Job step created
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2w80gz76vf57ljk-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2w80gz76vf57ljk-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=13 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-2w80gz76vf57ljk --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/85d8b58c155fe40d660b4af71b325e621820f2299c46c135041736fc10a8a317/cgroup.procs
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/85d8b58c155fe40d660b4af71b325e621820f2299c46c135041736fc10a8a317/memory.stat
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4743168 rss
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/85d8b58c155fe40d660b4af71b325e621820f2299c46c135041736fc10a8a317/cpuacct.stat
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/85d8b58c155fe40d660b4af71b325e621820f2299c46c135041736fc10a8a317/cpuset.cpus
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/85d8b58c155fe40d660b4af71b325e621820f2299c46c135041736fc10a8a317/blkio.io_service_bytes
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Mounting task input collection to get interval_list
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Interval_List collection mounted at /keep/d6cf1afdecfe2ae2261292a571a3759b+94
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Total chunk length is 15618872
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Have 1 intervals
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.16.26464','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-48tg8mv5801snwy-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-48tg8mv5801snwy-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-48tg8mv5801snwy-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=16 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-48tg8mv5801snwy --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:39:06 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 9 done, 8 running, 192 todo
2016-12-07_01:39:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:39:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:39:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:39:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:39:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:39:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Have 5400 potential reusable task outputs
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 killing orphaned srun process 23358 (task not in slurm queue, no stderr received in last 80s)
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 sending 2x signal 2 to pid 23358
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 killing orphaned srun process 23301 (task not in slurm queue, no stderr received in last 80s)
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 sending 2x signal 2 to pid 23301
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46034944 rss
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: cpu 0.6800 user 0.2100 sys 32 cpus -- interval 10.0001 seconds 0.6400 user 0.1900 sys
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr crunchstat: net:eth0 33951 tx 3236079 rx
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Have 5400 potentially reusable tasks
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr Creating task to process interval: [10:36622778-52241649]
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 stderr No reusable JobTask matched key parameters [u'111_of_200', u'94eedabc31fa41e0ae0f2bed3edca749+14699', '10:36622778-52241649', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 child 23301 on humgen-05-15.4 exit 1 success=
2016-12-07_01:39:10 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:39:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 failure (#1, temporary) after 98 seconds
2016-12-07_01:39:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 4 task output (0 bytes): 
2016-12-07_01:39:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 child 23358 on humgen-05-15.8 exit 1 success=
2016-12-07_01:39:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:39:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 failure (#1, temporary) after 99 seconds
2016-12-07_01:39:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 8 task output (0 bytes): 
2016-12-07_01:39:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 child 25497 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:39:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 success in 39 seconds
2016-12-07_01:39:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 13 task output (0 bytes): 
2016-12-07_01:39:14 z8ta6-8i9sb-r52unvxj6avgrio 19831  3 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:39:20 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:39:20 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 10 done, 5 running, 195 todo
2016-12-07_01:39:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 job_task z8ta6-ot0gb-wjeut4ryony5g0q
2016-12-07_01:39:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 child 26932 started on humgen-05-15.4
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 job_task z8ta6-ot0gb-2imne3wl48zosm9
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 child 26972 started on humgen-05-15.8
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 job_task z8ta6-ot0gb-v4gd5wmxbh7n2wd
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 child 26976 started on humgen-05-15.2
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.17.26932','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-wjeut4ryony5g0q-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-wjeut4ryony5g0q-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-wjeut4ryony5g0q-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=17 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-wjeut4ryony5g0q --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.18.26972','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2imne3wl48zosm9-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2imne3wl48zosm9-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2imne3wl48zosm9-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=18 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2imne3wl48zosm9 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr srun: Job step created
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8b8dww9azwpsvqq-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8b8dww9azwpsvqq-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=12 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-8b8dww9azwpsvqq --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7f560d63f3dd95655ec00649a05e3538d4b12bb2b580401f6f608a032277693f/cgroup.procs
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/7f560d63f3dd95655ec00649a05e3538d4b12bb2b580401f6f608a032277693f/memory.stat
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5189632 rss
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7f560d63f3dd95655ec00649a05e3538d4b12bb2b580401f6f608a032277693f/cpuacct.stat
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/7f560d63f3dd95655ec00649a05e3538d4b12bb2b580401f6f608a032277693f/cpuset.cpus
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/7f560d63f3dd95655ec00649a05e3538d4b12bb2b580401f6f608a032277693f/blkio.io_service_bytes
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Mounting task input collection to get interval_list
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Interval_List collection mounted at /keep/11837032d32bcfff7055a484f9c0ddb8+94
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Total chunk length is 15618872
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Have 1 intervals
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Have 5400 potential reusable task outputs
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Have 5400 potentially reusable tasks
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Creating task to process interval: [10:21003906-36622777]
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr No reusable JobTask matched key parameters [u'110_of_200', u'e20f5f00b25cd7e53f799770dbe9e308+14700', '10:21003906-36622777', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 stderr Ending task 1 successfully
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr srun: Job step created
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-dwe3asp6mogvgwm-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dwe3asp6mogvgwm-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=14 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-dwe3asp6mogvgwm --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.19.26976','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-v4gd5wmxbh7n2wd-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-v4gd5wmxbh7n2wd-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-v4gd5wmxbh7n2wd-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=19 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-v4gd5wmxbh7n2wd --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 child 24879 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 success in 58 seconds
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831 12 task output (0 bytes): 
2016-12-07_01:39:21 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 killing orphaned srun process 23828 (task not in slurm queue, no stderr received in last 76s)
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 sending 2x signal 2 to pid 23828
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 11 done, 7 running, 193 todo
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 job_task z8ta6-ot0gb-56iqz9bm59n73r5
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 child 27196 started on humgen-05-15.5
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.20.27196','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-56iqz9bm59n73r5-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-56iqz9bm59n73r5-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-56iqz9bm59n73r5-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=20 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-56iqz9bm59n73r5 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/93028149223f8a19fe0694dac47a9d15f2f56922f5f6be06fb3da71683a79c41/cgroup.procs
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/93028149223f8a19fe0694dac47a9d15f2f56922f5f6be06fb3da71683a79c41/memory.stat
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 765952 rss
2016-12-07_01:39:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/93028149223f8a19fe0694dac47a9d15f2f56922f5f6be06fb3da71683a79c41/cpuacct.stat
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/93028149223f8a19fe0694dac47a9d15f2f56922f5f6be06fb3da71683a79c41/cpuset.cpus
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: cpu 0.0000 user 0.0000 sys 32 cpus
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/93028149223f8a19fe0694dac47a9d15f2f56922f5f6be06fb3da71683a79c41/blkio.io_service_bytes
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Mounting task input collection to get interval_list
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Interval_List collection mounted at /keep/fe3ded28e7bf89d25f8d4e08ecc0d6c5+94
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Total chunk length is 15618872
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Have 1 intervals
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 child 23828 on humgen-05-15.1 exit 1 success=
2016-12-07_01:39:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:39:30 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 failure (#1, temporary) after 100 seconds
2016-12-07_01:39:30 z8ta6-8i9sb-r52unvxj6avgrio 19831 9 task output (0 bytes): 
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 11 done, 7 running, 193 todo
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Have 5400 potential reusable task outputs
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46034944 rss
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: cpu 0.6600 user 0.2300 sys 32 cpus -- interval 10.0001 seconds 0.6600 user 0.2300 sys
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr crunchstat: net:eth0 41319 tx 3249000 rx
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Have 5400 potentially reusable tasks
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Creating task to process interval: [10:52241650-67860521]
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr No reusable JobTask matched key parameters [u'112_of_200', u'9e1996fed17b6a2fca279a862266ce9d+14700', '10:52241650-67860521', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:39:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 stderr Ending task 1 successfully
2016-12-07_01:39:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 child 25848 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:39:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 success in 47 seconds
2016-12-07_01:39:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 14 task output (0 bytes): 
2016-12-07_01:39:32 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 12 done, 6 running, 194 todo
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 job_task z8ta6-ot0gb-ew3rr0cqwgqx9xv
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 child 27517 started on humgen-05-15.6
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 job_task z8ta6-ot0gb-nbruef2k5v3riu4
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 child 27537 started on humgen-05-15.1
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.21.27517','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ew3rr0cqwgqx9xv-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ew3rr0cqwgqx9xv-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ew3rr0cqwgqx9xv-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=21 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ew3rr0cqwgqx9xv --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-wjeut4ryony5g0q-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-wjeut4ryony5g0q-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=17 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-wjeut4ryony5g0q --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/001b9348a9bdec4c6ffa344d81e8b9273618d99a848a4e46a4839c9a478f045a/cgroup.procs
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/001b9348a9bdec4c6ffa344d81e8b9273618d99a848a4e46a4839c9a478f045a/memory.stat
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5341184 rss
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/001b9348a9bdec4c6ffa344d81e8b9273618d99a848a4e46a4839c9a478f045a/cpuacct.stat
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/001b9348a9bdec4c6ffa344d81e8b9273618d99a848a4e46a4839c9a478f045a/cpuset.cpus
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/001b9348a9bdec4c6ffa344d81e8b9273618d99a848a4e46a4839c9a478f045a/blkio.io_service_bytes
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Mounting task input collection to get interval_list
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Interval_List collection mounted at /keep/e05c9a56f87bbb0e87bd78edfb2138bb+94
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Total chunk length is 15618872
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Have 1 intervals
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.22.27537','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-nbruef2k5v3riu4-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-nbruef2k5v3riu4-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-nbruef2k5v3riu4-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=22 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-nbruef2k5v3riu4 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:39:39 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 12 done, 8 running, 192 todo
2016-12-07_01:39:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:39:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:39:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:39:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:39:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: blkio:0:0 0 write 12527 read -- interval 10.0000 seconds 0 write 12527 read
2016-12-07_01:39:43 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:39:43 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: cpu 0.5000 user 0.1700 sys 32 cpus -- interval 10.0001 seconds 0.4600 user 0.1600 sys
2016-12-07_01:39:43 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr crunchstat: net:eth0 21877 tx 238283 rx
2016-12-07_01:39:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Have 5400 potential reusable task outputs
2016-12-07_01:39:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:39:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:39:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Have 5400 potentially reusable tasks
2016-12-07_01:39:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Creating task to process interval: [10:99098266-114717137]
2016-12-07_01:39:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr No reusable JobTask matched key parameters [u'115_of_200', u'5601c60d4022448c1a998597fc7d32dc+14700', '10:99098266-114717137', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:39:44 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 stderr Ending task 1 successfully
2016-12-07_01:39:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 child 26932 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:39:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 success in 25 seconds
2016-12-07_01:39:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 17 task output (0 bytes): 
2016-12-07_01:39:45 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:39:51 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:39:51 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 13 done, 7 running, 193 todo
2016-12-07_01:39:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 job_task z8ta6-ot0gb-qcyjnrnt7w80tgq
2016-12-07_01:39:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 child 27875 started on humgen-05-15.4
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.23.27875','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qcyjnrnt7w80tgq-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qcyjnrnt7w80tgq-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qcyjnrnt7w80tgq-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=23 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-qcyjnrnt7w80tgq --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-56iqz9bm59n73r5-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-56iqz9bm59n73r5-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=20 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-56iqz9bm59n73r5 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8b3e1b2955752e197affd97873a6184adda0a00edbce7b11d367a1d7dd0a2b74/cgroup.procs
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8b3e1b2955752e197affd97873a6184adda0a00edbce7b11d367a1d7dd0a2b74/memory.stat
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4755456 rss
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8b3e1b2955752e197affd97873a6184adda0a00edbce7b11d367a1d7dd0a2b74/cpuacct.stat
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8b3e1b2955752e197affd97873a6184adda0a00edbce7b11d367a1d7dd0a2b74/cpuset.cpus
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8b3e1b2955752e197affd97873a6184adda0a00edbce7b11d367a1d7dd0a2b74/blkio.io_service_bytes
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Mounting task input collection to get interval_list
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Interval_List collection mounted at /keep/d9aa702cd7b3a1e6b6dec54b4a3d8e35+94
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Total chunk length is 15618872
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Have 1 intervals
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:39:52 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 13 done, 8 running, 192 todo
2016-12-07_01:39:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Have 5400 potential reusable task outputs
2016-12-07_01:39:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:39:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:39:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Have 5400 potentially reusable tasks
2016-12-07_01:39:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Creating task to process interval: [11:10300135-25919006]
2016-12-07_01:39:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr No reusable JobTask matched key parameters [u'118_of_200', u'79b22e2a5c2a107fd090c78ed23ff2e7+14700', '11:10300135-25919006', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:39:54 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 stderr Ending task 1 successfully
2016-12-07_01:39:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:39:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:39:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr srun: Job step created
2016-12-07_01:39:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 child 27196 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:39:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 success in 27 seconds
2016-12-07_01:39:55 z8ta6-8i9sb-r52unvxj6avgrio 19831 20 task output (0 bytes): 
2016-12-07_01:39:55 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:40:01 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 notice: task is not in slurm queue but srun process 26154 has not exited
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 14 done, 7 running, 193 todo
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 job_task z8ta6-ot0gb-puq0tpp4073idhj
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 child 28255 started on humgen-05-15.5
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.24.28255','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-puq0tpp4073idhj-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-puq0tpp4073idhj-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-puq0tpp4073idhj-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=24 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-puq0tpp4073idhj --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-48tg8mv5801snwy-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-48tg8mv5801snwy-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=16 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-48tg8mv5801snwy --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8793c835853e5325dfdafac198bd571c251467761ab5432cd0c2cc0299511359/cgroup.procs
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8793c835853e5325dfdafac198bd571c251467761ab5432cd0c2cc0299511359/memory.stat
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7462912 rss
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8793c835853e5325dfdafac198bd571c251467761ab5432cd0c2cc0299511359/cpuacct.stat
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8793c835853e5325dfdafac198bd571c251467761ab5432cd0c2cc0299511359/cpuset.cpus
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8793c835853e5325dfdafac198bd571c251467761ab5432cd0c2cc0299511359/blkio.io_service_bytes
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Mounting task input collection to get interval_list
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Interval_List collection mounted at /keep/8b0a6de6be0a7b3dd7360336477e02fd+94
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Total chunk length is 15618872
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Have 1 intervals
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:40:02 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 14 done, 8 running, 192 todo
2016-12-07_01:40:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:40:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:40:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:40:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:40:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:40:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 25948160 rss
2016-12-07_01:40:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: cpu 0.4900 user 0.1900 sys 32 cpus -- interval 10.0002 seconds 0.4600 user 0.1900 sys
2016-12-07_01:40:05 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr crunchstat: net:eth0 30325 tx 3237833 rx
2016-12-07_01:40:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Have 5400 potential reusable task outputs
2016-12-07_01:40:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:40:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:40:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Have 5400 potentially reusable tasks
2016-12-07_01:40:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Creating task to process interval: [10:83479394-99098265]
2016-12-07_01:40:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr No reusable JobTask matched key parameters [u'114_of_200', u'c3bc3b07edcf5d2b7b9bfb2b5154d56f+14700', '10:83479394-99098265', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:40:06 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 stderr Ending task 1 successfully
2016-12-07_01:40:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 child 26464 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:40:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 success in 61 seconds
2016-12-07_01:40:07 z8ta6-8i9sb-r52unvxj6avgrio 19831 16 task output (0 bytes): 
2016-12-07_01:40:07 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:40:13 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:40:13 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 15 done, 7 running, 193 todo
2016-12-07_01:40:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 job_task z8ta6-ot0gb-9sx9kmd39pedd7f
2016-12-07_01:40:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 child 28915 started on humgen-05-15.7
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qcyjnrnt7w80tgq-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qcyjnrnt7w80tgq-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=23 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-qcyjnrnt7w80tgq --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ceacaf3db59f6b7dce0fc39a8d93f002108cb7c64ae9d522e1dcb294248b9a47/cgroup.procs
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/ceacaf3db59f6b7dce0fc39a8d93f002108cb7c64ae9d522e1dcb294248b9a47/memory.stat
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5660672 rss
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ceacaf3db59f6b7dce0fc39a8d93f002108cb7c64ae9d522e1dcb294248b9a47/cpuacct.stat
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/ceacaf3db59f6b7dce0fc39a8d93f002108cb7c64ae9d522e1dcb294248b9a47/cpuset.cpus
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/ceacaf3db59f6b7dce0fc39a8d93f002108cb7c64ae9d522e1dcb294248b9a47/blkio.io_service_bytes
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Mounting task input collection to get interval_list
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Interval_List collection mounted at /keep/b34c79e916b3f4868c677e831b11c26d+94
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Total chunk length is 15618872
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Have 1 intervals
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.25.28915','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9sx9kmd39pedd7f-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-9sx9kmd39pedd7f-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9sx9kmd39pedd7f-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=25 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-9sx9kmd39pedd7f --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:40:14 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 15 done, 8 running, 192 todo
2016-12-07_01:40:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:40:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:40:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:40:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:40:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:40:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24084480 rss
2016-12-07_01:40:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: cpu 0.4800 user 0.1500 sys 32 cpus -- interval 10.0000 seconds 0.4400 user 0.1400 sys
2016-12-07_01:40:17 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr crunchstat: net:eth0 21919 tx 238193 rx
2016-12-07_01:40:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Have 5400 potential reusable task outputs
2016-12-07_01:40:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:40:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:40:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Have 5400 potentially reusable tasks
2016-12-07_01:40:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Creating task to process interval: [11:41537879-57156750]
2016-12-07_01:40:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr No reusable JobTask matched key parameters [u'120_of_200', u'4253ced78675bc039ec63c878e7f8d99+14700', '11:41537879-57156750', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:40:18 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 stderr Ending task 1 successfully
2016-12-07_01:40:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr srun: Job step created
2016-12-07_01:40:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:40:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 child 27875 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:40:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 success in 28 seconds
2016-12-07_01:40:19 z8ta6-8i9sb-r52unvxj6avgrio 19831 23 task output (0 bytes): 
2016-12-07_01:40:19 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 16 done, 7 running, 193 todo
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 job_task z8ta6-ot0gb-i8elr66xa9l371t
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 child 29253 started on humgen-05-15.4
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.26.29253','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i8elr66xa9l371t-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-i8elr66xa9l371t-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i8elr66xa9l371t-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=26 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-i8elr66xa9l371t --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2imne3wl48zosm9-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2imne3wl48zosm9-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=18 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-2imne3wl48zosm9 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/c4cb93844703c941129f2e80c71d9e02b122e5d17fcfb1750f219b9a610e862b/cgroup.procs
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/c4cb93844703c941129f2e80c71d9e02b122e5d17fcfb1750f219b9a610e862b/memory.stat
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4341760 rss
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/c4cb93844703c941129f2e80c71d9e02b122e5d17fcfb1750f219b9a610e862b/cpuacct.stat
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/c4cb93844703c941129f2e80c71d9e02b122e5d17fcfb1750f219b9a610e862b/cpuset.cpus
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/c4cb93844703c941129f2e80c71d9e02b122e5d17fcfb1750f219b9a610e862b/blkio.io_service_bytes
2016-12-07_01:40:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Mounting task input collection to get interval_list
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Interval_List collection mounted at /keep/359a96b521b56e8d8e36b57450fa80ac+94
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Total chunk length is 15618872
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Have 1 intervals
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:40:26 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 16 done, 8 running, 192 todo
2016-12-07_01:40:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Have 5400 potential reusable task outputs
2016-12-07_01:40:28 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Have 5400 potentially reusable tasks
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Creating task to process interval: [10:114717138-130336009]
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr No reusable JobTask matched key parameters [u'116_of_200', u'6365525c0ee094fee362794dbbab6a30+14700', '10:114717138-130336009', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr Ending task 1 successfully
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 stderr crunchstat: blkio:0:0 0 write 12528 read -- interval 10.0000 seconds 0 write 12528 read
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 child 26972 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 success in 68 seconds
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831 18 task output (0 bytes): 
2016-12-07_01:40:29 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:40:36 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:40:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 killing orphaned srun process 26154 (task not in slurm queue, no stderr received in last 75s)
2016-12-07_01:40:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 sending 2x signal 2 to pid 26154
2016-12-07_01:40:36 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 notice: task is not in slurm queue but srun process 26976 has not exited
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 17 done, 7 running, 193 todo
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 job_task z8ta6-ot0gb-kzj7ro4tvmzyqig
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 child 29621 started on humgen-05-15.8
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.27.29621','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kzj7ro4tvmzyqig-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-kzj7ro4tvmzyqig-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kzj7ro4tvmzyqig-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=27 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-kzj7ro4tvmzyqig --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-9sx9kmd39pedd7f-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9sx9kmd39pedd7f-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=25 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-9sx9kmd39pedd7f --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/fca094c6633620a10dcf0c3c8bca2785aef71538ea491ed8a6d686c82f7461e0/cgroup.procs
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/fca094c6633620a10dcf0c3c8bca2785aef71538ea491ed8a6d686c82f7461e0/memory.stat
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5128192 rss
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/fca094c6633620a10dcf0c3c8bca2785aef71538ea491ed8a6d686c82f7461e0/cpuacct.stat
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/fca094c6633620a10dcf0c3c8bca2785aef71538ea491ed8a6d686c82f7461e0/cpuset.cpus
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/fca094c6633620a10dcf0c3c8bca2785aef71538ea491ed8a6d686c82f7461e0/blkio.io_service_bytes
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Mounting task input collection to get interval_list
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Interval_List collection mounted at /keep/bbfdffd38f1432cef552ced110ab3a90+94
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Total chunk length is 15618872
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Have 1 intervals
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 child 26154 on humgen-05-15.3 exit 1 success=
2016-12-07_01:40:37 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:40:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 failure (#1, temporary) after 101 seconds
2016-12-07_01:40:39 z8ta6-8i9sb-r52unvxj6avgrio 19831 15 task output (0 bytes): 
2016-12-07_01:40:39 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 17 done, 7 running, 193 todo
2016-12-07_01:40:39 z8ta6-8i9sb-r52unvxj6avgrio 19831  Every node has failed -- giving up
2016-12-07_01:40:39 z8ta6-8i9sb-r52unvxj6avgrio 19831  wait for last 7 children to finish
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Have 5400 potential reusable task outputs
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Have 5400 potentially reusable tasks
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Creating task to process interval: [11:72775623-88394494]
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr No reusable JobTask matched key parameters [u'122_of_200', u'af62e8edc576e4baf02413c2134c9270+14700', '11:72775623-88394494', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 49074176 rss
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: cpu 0.7200 user 0.2100 sys 32 cpus -- interval 10.0001 seconds 0.6900 user 0.1900 sys
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr crunchstat: net:eth0 37310 tx 3247130 rx
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 stderr Ending task 1 successfully
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr srun: Job step created
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 child 28915 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:40:40 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 success in 27 seconds
2016-12-07_01:40:41 z8ta6-8i9sb-r52unvxj6avgrio 19831 25 task output (0 bytes): 
2016-12-07_01:40:41 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-nbruef2k5v3riu4-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-nbruef2k5v3riu4-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=22 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-nbruef2k5v3riu4 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/1a862586c229d65ef878ec929043477c84b54d5271e7c57abc2cbb62defb7b16/cgroup.procs
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/1a862586c229d65ef878ec929043477c84b54d5271e7c57abc2cbb62defb7b16/memory.stat
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5693440 rss
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/1a862586c229d65ef878ec929043477c84b54d5271e7c57abc2cbb62defb7b16/cpuacct.stat
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/1a862586c229d65ef878ec929043477c84b54d5271e7c57abc2cbb62defb7b16/cpuset.cpus
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/1a862586c229d65ef878ec929043477c84b54d5271e7c57abc2cbb62defb7b16/blkio.io_service_bytes
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Mounting task input collection to get interval_list
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Interval_List collection mounted at /keep/a19791935337560ca00961b9d0c50277+92
2016-12-07_01:40:47 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Total chunk length is 15618872
2016-12-07_01:40:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:40:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Have 1 intervals
2016-12-07_01:40:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:40:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:40:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:40:48 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:40:48 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 18 done, 6 running, 194 todo
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 notice: task is not in slurm queue but srun process 27517 has not exited
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24084480 rss
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: cpu 0.4800 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.4400 user 0.1700 sys
2016-12-07_01:40:51 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr crunchstat: net:eth0 21877 tx 238348 rx
2016-12-07_01:40:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Have 5400 potential reusable task outputs
2016-12-07_01:40:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:40:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:40:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Have 5400 potentially reusable tasks
2016-12-07_01:40:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Creating task to process interval: [1:156188721-171807592]
2016-12-07_01:40:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr No reusable JobTask matched key parameters [u'11_of_200', u'0407d47186859c17195bd2efb49797bb+14616', '1:156188721-171807592', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:40:52 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 stderr Ending task 1 successfully
2016-12-07_01:40:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 child 27537 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:40:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 success in 74 seconds
2016-12-07_01:40:53 z8ta6-8i9sb-r52unvxj6avgrio 19831 22 task output (0 bytes): 
2016-12-07_01:40:53 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr srun: Job step created
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ew3rr0cqwgqx9xv-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ew3rr0cqwgqx9xv-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=21 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ew3rr0cqwgqx9xv --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8001da9a2c9f6c3273d8dc4f7e3e7b7be71cc671c2a79e7f22b3cee6a64c7327/cgroup.procs
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8001da9a2c9f6c3273d8dc4f7e3e7b7be71cc671c2a79e7f22b3cee6a64c7327/cpuacct.stat
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr crunchstat: cpu 0.0000 user 0.0000 sys 0 cpus
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8001da9a2c9f6c3273d8dc4f7e3e7b7be71cc671c2a79e7f22b3cee6a64c7327/blkio.io_service_bytes
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Mounting task input collection to get interval_list
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Interval_List collection mounted at /keep/dbba1077cec452fe00018827674a6082+94
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Total chunk length is 15618872
2016-12-07_01:40:59 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:41:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Have 1 intervals
2016-12-07_01:41:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:41:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:41:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:41:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:41:00 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:41:00 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 19 done, 5 running, 195 todo
2016-12-07_01:41:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Have 5400 potential reusable task outputs
2016-12-07_01:41:01 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Have 5400 potentially reusable tasks
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Creating task to process interval: [11:25919007-41537878]
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr No reusable JobTask matched key parameters [u'119_of_200', u'e715710b28adf5479c2b4ee6a270292a+14700', '11:25919007-41537878', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 stderr Ending task 1 successfully
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 child 27517 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 success in 83 seconds
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831 21 task output (0 bytes): 
2016-12-07_01:41:02 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr srun: Job step created
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-v4gd5wmxbh7n2wd-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-v4gd5wmxbh7n2wd-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=19 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-v4gd5wmxbh7n2wd --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d3e03075426d1e6345f09c86746a516856d72eaa159789605df9562f37c14661/cgroup.procs
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d3e03075426d1e6345f09c86746a516856d72eaa159789605df9562f37c14661/cpuacct.stat
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: cpu 0.0200 user 0.0000 sys 0 cpus
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d3e03075426d1e6345f09c86746a516856d72eaa159789605df9562f37c14661/blkio.io_service_bytes
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Mounting task input collection to get interval_list
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Interval_List collection mounted at /keep/76a230c91cc5d0053b75306d58906b2a+94
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Total chunk length is 15498872
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Have 1 intervals
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:41:09 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 notice: task is not in slurm queue but srun process 28255 has not exited
2016-12-07_01:41:10 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 20 done, 4 running, 196 todo
2016-12-07_01:41:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Have 5400 potential reusable task outputs
2016-12-07_01:41:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:41:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:41:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Have 5400 potentially reusable tasks
2016-12-07_01:41:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Creating task to process interval: [10:130336010-135534747 11:1-10300134]
2016-12-07_01:41:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr No reusable JobTask matched key parameters [u'117_of_200', u'a986cd76d508917b0b96bd60c6107961+14704', '10:130336010-135534747 11:1-10300134', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:41:12 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr Ending task 1 successfully
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 stderr crunchstat: blkio:0:0 0 write 12567 read -- interval 10.0000 seconds 0 write 12567 read
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 child 26976 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 success in 112 seconds
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831 19 task output (0 bytes): 
2016-12-07_01:41:13 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr srun: Job step created
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-i8elr66xa9l371t-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i8elr66xa9l371t-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=26 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-i8elr66xa9l371t --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/29c357241426104c174c12a37a5bff3426e9bac99686936b3661adf82da77658/cgroup.procs
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/29c357241426104c174c12a37a5bff3426e9bac99686936b3661adf82da77658/memory.stat
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4452352 rss
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/29c357241426104c174c12a37a5bff3426e9bac99686936b3661adf82da77658/cpuacct.stat
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/29c357241426104c174c12a37a5bff3426e9bac99686936b3661adf82da77658/cpuset.cpus
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/29c357241426104c174c12a37a5bff3426e9bac99686936b3661adf82da77658/blkio.io_service_bytes
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Mounting task input collection to get interval_list
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Interval_List collection mounted at /keep/ddb0ad35fd81534c2371a454b4e54268+94
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Total chunk length is 15618872
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Have 1 intervals
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:41:20 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 21 done, 3 running, 197 todo
2016-12-07_01:41:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:41:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:41:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:41:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:41:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: blkio:0:0 0 write 12527 read -- interval 10.0000 seconds 0 write 12527 read
2016-12-07_01:41:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:41:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: cpu 0.5200 user 0.1300 sys 32 cpus -- interval 10.0002 seconds 0.5000 user 0.1300 sys
2016-12-07_01:41:23 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr crunchstat: net:eth0 21877 tx 238217 rx
2016-12-07_01:41:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Have 5400 potential reusable task outputs
2016-12-07_01:41:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:41:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:41:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Have 5400 potentially reusable tasks
2016-12-07_01:41:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Creating task to process interval: [11:88394495-104013366]
2016-12-07_01:41:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr No reusable JobTask matched key parameters [u'123_of_200', u'b28694acffb44b65c84fe09c422e709c+14700', '11:88394495-104013366', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:41:24 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 stderr Ending task 1 successfully
2016-12-07_01:41:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr srun: Job step created
2016-12-07_01:41:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 child 29253 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:41:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 success in 60 seconds
2016-12-07_01:41:25 z8ta6-8i9sb-r52unvxj6avgrio 19831 26 task output (0 bytes): 
2016-12-07_01:41:25 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-kzj7ro4tvmzyqig-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kzj7ro4tvmzyqig-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=27 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-kzj7ro4tvmzyqig --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3e4d3a53c7f9de3d46d41d80c6b7aa725521d147c31d7451c24140dc1bbbe822/cgroup.procs
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/3e4d3a53c7f9de3d46d41d80c6b7aa725521d147c31d7451c24140dc1bbbe822/memory.stat
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5922816 rss
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3e4d3a53c7f9de3d46d41d80c6b7aa725521d147c31d7451c24140dc1bbbe822/cpuacct.stat
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/3e4d3a53c7f9de3d46d41d80c6b7aa725521d147c31d7451c24140dc1bbbe822/cpuset.cpus
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/3e4d3a53c7f9de3d46d41d80c6b7aa725521d147c31d7451c24140dc1bbbe822/blkio.io_service_bytes
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Mounting task input collection to get interval_list
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Interval_List collection mounted at /keep/b4389106228a3052a1940c08eb93c60e+94
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Total chunk length is 15618872
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Have 1 intervals
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:41:31 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:41:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:41:32 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:41:32 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 22 done, 2 running, 198 todo
2016-12-07_01:41:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Have 5400 potential reusable task outputs
2016-12-07_01:41:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:41:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:41:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Have 5400 potentially reusable tasks
2016-12-07_01:41:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Creating task to process interval: [11:104013367-119632238]
2016-12-07_01:41:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr No reusable JobTask matched key parameters [u'124_of_200', u'd368badda25753b2dc89a57d77b90d16+14700', '11:104013367-119632238', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:41:34 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 stderr Ending task 1 successfully
2016-12-07_01:41:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 child 29621 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:41:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 success in 58 seconds
2016-12-07_01:41:35 z8ta6-8i9sb-r52unvxj6avgrio 19831 27 task output (0 bytes): 
2016-12-07_01:41:35 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr srun: Job step created
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-puq0tpp4073idhj-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-puq0tpp4073idhj-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=24 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-puq0tpp4073idhj --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/471ad0e90142ef706e4292881d2a895380ebf1aa1fc6c732aae8455693db1a53/cgroup.procs
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/471ad0e90142ef706e4292881d2a895380ebf1aa1fc6c732aae8455693db1a53/memory.stat
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4063232 rss
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/471ad0e90142ef706e4292881d2a895380ebf1aa1fc6c732aae8455693db1a53/cpuacct.stat
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/471ad0e90142ef706e4292881d2a895380ebf1aa1fc6c732aae8455693db1a53/cpuset.cpus
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/471ad0e90142ef706e4292881d2a895380ebf1aa1fc6c732aae8455693db1a53/blkio.io_service_bytes
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Mounting task input collection to get interval_list
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Interval_List collection mounted at /keep/dcd0caa42cf81cfae2c2a7a7d6a5503a+94
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Total chunk length is 15618872
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Have 1 intervals
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:41:42 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 23 done, 1 running, 199 todo
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Have 5400 potential reusable task outputs
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Have 5400 potentially reusable tasks
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Creating task to process interval: [11:57156751-72775622]
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr No reusable JobTask matched key parameters [u'121_of_200', u'7770dae0a3517e6dca33c49e54e80074+14704', '11:57156751-72775622', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:41:45 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr Ending task 1 successfully
2016-12-07_01:41:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 37175296 rss
2016-12-07_01:41:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: cpu 0.8100 user 0.2000 sys 32 cpus -- interval 10.0002 seconds 0.7800 user 0.2000 sys
2016-12-07_01:41:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 stderr crunchstat: net:eth0 39219 tx 3252359 rx
2016-12-07_01:41:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 child 28255 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:41:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 success in 104 seconds
2016-12-07_01:41:46 z8ta6-8i9sb-r52unvxj6avgrio 19831 24 task output (0 bytes): 
2016-12-07_01:41:46 z8ta6-8i9sb-r52unvxj6avgrio 19831  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:41:53 z8ta6-8i9sb-r52unvxj6avgrio 19831  Got 1 new tasks from API server.
2016-12-07_01:41:53 z8ta6-8i9sb-r52unvxj6avgrio 19831  status: 24 done, 0 running, 200 todo
2016-12-07_01:41:53 z8ta6-8i9sb-r52unvxj6avgrio 19831  collate
2016-12-07_01:41:53 z8ta6-8i9sb-r52unvxj6avgrio 19831  collated output manifest text to send to API server is 0 bytes with access tokens
2016-12-07_01:41:53 z8ta6-8i9sb-r52unvxj6avgrio 19831  job output d41d8cd98f00b204e9800998ecf8427e+0
2016-12-07_01:41:56 z8ta6-8i9sb-r52unvxj6avgrio 32237  running from /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/arvados-cli-0.1.20161017193526/bin/crunch-job with arvados-cli Gem version(s) 0.1.20161017193526, 0.1.20160503204200, 0.1.20151207150126, 0.1.20151023190001
2016-12-07_01:41:56 z8ta6-8i9sb-r52unvxj6avgrio 32237  check slurm allocation
2016-12-07_01:41:56 z8ta6-8i9sb-r52unvxj6avgrio 32237  node humgen-05-15 - 8 slots
2016-12-07_01:41:57 z8ta6-8i9sb-r52unvxj6avgrio 32237  start
2016-12-07_01:41:57 z8ta6-8i9sb-r52unvxj6avgrio 32237  clean work dirs: start
2016-12-07_01:41:57 z8ta6-8i9sb-r52unvxj6avgrio 32237  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk "(index(\\$3, \\"$CRUNCH_TMP\\") == 1){print \\$3}" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']
2016-12-07_01:41:58 z8ta6-8i9sb-r52unvxj6avgrio 32237  clean work dirs: exit 0
2016-12-07_01:41:58 z8ta6-8i9sb-r52unvxj6avgrio 32237  Install docker image 9289c4f1fce1d1dedf9cc7876c33f936+671
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  docker image hash is 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  load docker image: start
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  stderr starting: ['srun','--nodelist=humgen-05-15','/bin/bash','-o','pipefail','-ec',' if /usr/bin/docker images -q --no-trunc --all | grep -xF 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 >/dev/null; then     exit 0 fi declare -a exit_codes=("${PIPESTATUS[@]}") if [ 0 != "${exit_codes[0]}" ]; then    exit "${exit_codes[0]}"  # `docker images` failed elif [ 1 != "${exit_codes[1]}" ]; then    exit "${exit_codes[1]}"  # `grep` encountered an error else    # Everything worked fine, but grep didn\'t find the image on this host.    arv-get 9289c4f1fce1d1dedf9cc7876c33f936\\+671\\/571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3\\.tar | /usr/bin/docker load fi ']
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  load docker image: exit 0
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  check --memory-swap feature: start
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  stderr starting: ['srun','--nodes=1','/usr/bin/docker','run','--help']
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  check --memory-swap feature: exit 0
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  check whether default user is UID 0: start
2016-12-07_01:41:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  stderr starting: ['srun','--nodes=1','/bin/sh','-ec','/usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21  571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 id --user']
2016-12-07_01:42:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  check whether default user is UID 0: exit 0
2016-12-07_01:42:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  Container will run with default user
2016-12-07_01:42:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  Looking for version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 from repository jr17/hgi
2016-12-07_01:42:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  Using local repository '/var/lib/arvados/internal.git'
2016-12-07_01:42:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  Version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 is commit 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:42:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  run install script on all workers: start
2016-12-07_01:42:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio','sh','-c','mkdir -p /data/crunch-tmp/crunch-job/opt && cd /data/crunch-tmp/crunch-job && perl -']
2016-12-07_01:42:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  stderr Extracting archive: 4b8cf8e1274f8d418cb494d3ea978152
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237  run install script on all workers: exit 0
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237  script gatk-combinegvcfs.py
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237  script_version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237  script_parameters {"interval_lists_collection":"4a72f79fd2ed51ac692bc326fb168f37+10043","reference_collection":"1b447d1aa006cde57df03042736750b9+7894","inputs_collection":"b0dd82bb014b6ab8e64968ebdc07aad9+2566847"}
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237  runtime_constraints {"keep_cache_mb_per_task":19500,"max_tasks_per_node":8,"min_nodes":1,"docker_image":"mercury/gatk-3.6"}
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237  start level 0 with 1 slots
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 0 done, 0 running, 1 todo
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 job_task z8ta6-ot0gb-ox5yi0n6hx13v25
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 child 32476 started on humgen-05-15.1
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.0.32476','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (1 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ox5yi0n6hx13v25-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ox5yi0n6hx13v25-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ox5yi0n6hx13v25-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=0 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=0 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ox5yi0n6hx13v25 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 0 done, 1 running, 0 todo
2016-12-07_01:42:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ox5yi0n6hx13v25-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ox5yi0n6hx13v25-0.cid --sig-proxy --memory=250853295k --memory-swap=284407723k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=0 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=0 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ox5yi0n6hx13v25 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:42:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/02e702e5a88e74120970a21618816446b739878c914e083381a0f27ab7a05f4c/cgroup.procs
2016-12-07_01:42:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/02e702e5a88e74120970a21618816446b739878c914e083381a0f27ab7a05f4c/memory.stat
2016-12-07_01:42:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5779456 rss
2016-12-07_01:42:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/02e702e5a88e74120970a21618816446b739878c914e083381a0f27ab7a05f4c/cpuacct.stat
2016-12-07_01:42:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/02e702e5a88e74120970a21618816446b739878c914e083381a0f27ab7a05f4c/cpuset.cpus
2016-12-07_01:42:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:42:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/02e702e5a88e74120970a21618816446b739878c914e083381a0f27ab7a05f4c/blkio.io_service_bytes
2016-12-07_01:42:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:42:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Processing last stream
2016-12-07_01:42:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Finalising stream .
2016-12-07_01:42:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 100_of_200
2016-12-07_01:42:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 100_of_200
2016-12-07_01:42:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 101_of_200
2016-12-07_01:42:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 101_of_200
2016-12-07_01:42:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 102_of_200
2016-12-07_01:42:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 102_of_200
2016-12-07_01:42:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 103_of_200
2016-12-07_01:42:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 103_of_200
2016-12-07_01:42:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 104_of_200
2016-12-07_01:42:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 104_of_200
2016-12-07_01:42:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 105_of_200
2016-12-07_01:42:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 105_of_200
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 106_of_200
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 106_of_200
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 107_of_200
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 107_of_200
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 108_of_200
2016-12-07_01:42:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 159322112 rss
2016-12-07_01:42:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 4.5000 user 0.3500 sys 32 cpus -- interval 10.0001 seconds 4.4700 user 0.3300 sys
2016-12-07_01:42:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 231613 tx 3594258 rx
2016-12-07_01:42:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 108_of_200
2016-12-07_01:42:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 109_of_200
2016-12-07_01:42:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 109_of_200
2016-12-07_01:42:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 10_of_200
2016-12-07_01:42:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 10_of_200
2016-12-07_01:42:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 110_of_200
2016-12-07_01:42:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 110_of_200
2016-12-07_01:42:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 111_of_200
2016-12-07_01:42:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 111_of_200
2016-12-07_01:42:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 112_of_200
2016-12-07_01:42:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 112_of_200
2016-12-07_01:42:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 113_of_200
2016-12-07_01:42:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 113_of_200
2016-12-07_01:42:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 114_of_200
2016-12-07_01:42:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 114_of_200
2016-12-07_01:42:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 115_of_200
2016-12-07_01:42:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 115_of_200
2016-12-07_01:42:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 116_of_200
2016-12-07_01:42:16 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 116_of_200
2016-12-07_01:42:16 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 117_of_200
2016-12-07_01:42:16 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 117_of_200
2016-12-07_01:42:16 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 118_of_200
2016-12-07_01:42:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 118_of_200
2016-12-07_01:42:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 119_of_200
2016-12-07_01:42:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 119_of_200
2016-12-07_01:42:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 11_of_200
2016-12-07_01:42:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 11_of_200
2016-12-07_01:42:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 120_of_200
2016-12-07_01:42:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 120_of_200
2016-12-07_01:42:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 121_of_200
2016-12-07_01:42:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 121_of_200
2016-12-07_01:42:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 122_of_200
2016-12-07_01:42:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 122_of_200
2016-12-07_01:42:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 123_of_200
2016-12-07_01:42:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 123_of_200
2016-12-07_01:42:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 124_of_200
2016-12-07_01:42:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 124_of_200
2016-12-07_01:42:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 125_of_200
2016-12-07_01:42:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 125_of_200
2016-12-07_01:42:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 126_of_200
2016-12-07_01:42:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 126_of_200
2016-12-07_01:42:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 127_of_200
2016-12-07_01:42:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 127_of_200
2016-12-07_01:42:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 128_of_200
2016-12-07_01:42:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 128_of_200
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 177508352 rss
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 7.6400 user 0.5600 sys 32 cpus -- interval 10.0000 seconds 3.1400 user 0.2100 sys
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 757536 tx 4502419 rx -- interval 10.0000 seconds 525923 tx 908161 rx
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 129_of_200
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 129_of_200
2016-12-07_01:42:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 12_of_200
2016-12-07_01:42:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 12_of_200
2016-12-07_01:42:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 130_of_200
2016-12-07_01:42:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 130_of_200
2016-12-07_01:42:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 131_of_200
2016-12-07_01:42:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 131_of_200
2016-12-07_01:42:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 132_of_200
2016-12-07_01:42:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 132_of_200
2016-12-07_01:42:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 133_of_200
2016-12-07_01:42:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 133_of_200
2016-12-07_01:42:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 134_of_200
2016-12-07_01:42:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 134_of_200
2016-12-07_01:42:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 135_of_200
2016-12-07_01:42:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 135_of_200
2016-12-07_01:42:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 136_of_200
2016-12-07_01:42:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 136_of_200
2016-12-07_01:42:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 137_of_200
2016-12-07_01:42:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 137_of_200
2016-12-07_01:42:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 138_of_200
2016-12-07_01:42:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 138_of_200
2016-12-07_01:42:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 139_of_200
2016-12-07_01:42:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 139_of_200
2016-12-07_01:42:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 13_of_200
2016-12-07_01:42:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 13_of_200
2016-12-07_01:42:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 140_of_200
2016-12-07_01:42:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 140_of_200
2016-12-07_01:42:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 141_of_200
2016-12-07_01:42:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 141_of_200
2016-12-07_01:42:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 142_of_200
2016-12-07_01:42:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 142_of_200
2016-12-07_01:42:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 143_of_200
2016-12-07_01:42:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 143_of_200
2016-12-07_01:42:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 144_of_200
2016-12-07_01:42:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 144_of_200
2016-12-07_01:42:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 145_of_200
2016-12-07_01:42:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 145_of_200
2016-12-07_01:42:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 146_of_200
2016-12-07_01:42:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 146_of_200
2016-12-07_01:42:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 147_of_200
2016-12-07_01:42:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 147_of_200
2016-12-07_01:42:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 148_of_200
2016-12-07_01:42:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 148_of_200
2016-12-07_01:42:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 149_of_200
2016-12-07_01:42:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 149_of_200
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 14_of_200
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 182730752 rss
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 10.7800 user 0.7000 sys 32 cpus -- interval 10.0000 seconds 3.1400 user 0.1400 sys
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 1305739 tx 5440190 rx -- interval 10.0000 seconds 548203 tx 937771 rx
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 14_of_200
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 150_of_200
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 150_of_200
2016-12-07_01:42:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 151_of_200
2016-12-07_01:42:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 151_of_200
2016-12-07_01:42:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 152_of_200
2016-12-07_01:42:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 152_of_200
2016-12-07_01:42:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 153_of_200
2016-12-07_01:42:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 153_of_200
2016-12-07_01:42:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 154_of_200
2016-12-07_01:42:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 154_of_200
2016-12-07_01:42:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 155_of_200
2016-12-07_01:42:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 155_of_200
2016-12-07_01:42:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 156_of_200
2016-12-07_01:42:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 156_of_200
2016-12-07_01:42:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 157_of_200
2016-12-07_01:42:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 157_of_200
2016-12-07_01:42:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 158_of_200
2016-12-07_01:42:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 158_of_200
2016-12-07_01:42:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 159_of_200
2016-12-07_01:42:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 159_of_200
2016-12-07_01:42:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 15_of_200
2016-12-07_01:42:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 15_of_200
2016-12-07_01:42:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 160_of_200
2016-12-07_01:42:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 160_of_200
2016-12-07_01:42:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 161_of_200
2016-12-07_01:42:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 161_of_200
2016-12-07_01:42:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 162_of_200
2016-12-07_01:42:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 162_of_200
2016-12-07_01:42:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 163_of_200
2016-12-07_01:42:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:42:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:42:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:42:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 163_of_200
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 164_of_200
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 200642560 rss
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 12.9100 user 0.8200 sys 32 cpus -- interval 10.0000 seconds 2.1300 user 0.1200 sys
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 1685833 tx 6084424 rx -- interval 10.0000 seconds 380094 tx 644234 rx
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 164_of_200
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 165_of_200
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 165_of_200
2016-12-07_01:42:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 166_of_200
2016-12-07_01:42:43 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 166_of_200
2016-12-07_01:42:43 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 167_of_200
2016-12-07_01:42:43 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 167_of_200
2016-12-07_01:42:43 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 168_of_200
2016-12-07_01:42:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 168_of_200
2016-12-07_01:42:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 169_of_200
2016-12-07_01:42:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 169_of_200
2016-12-07_01:42:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 16_of_200
2016-12-07_01:42:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 16_of_200
2016-12-07_01:42:45 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 170_of_200
2016-12-07_01:42:45 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 170_of_200
2016-12-07_01:42:45 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 171_of_200
2016-12-07_01:42:45 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 171_of_200
2016-12-07_01:42:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 172_of_200
2016-12-07_01:42:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 172_of_200
2016-12-07_01:42:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 173_of_200
2016-12-07_01:42:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 173_of_200
2016-12-07_01:42:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 174_of_200
2016-12-07_01:42:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 174_of_200
2016-12-07_01:42:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 175_of_200
2016-12-07_01:42:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 175_of_200
2016-12-07_01:42:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 176_of_200
2016-12-07_01:42:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 176_of_200
2016-12-07_01:42:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 177_of_200
2016-12-07_01:42:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 177_of_200
2016-12-07_01:42:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 178_of_200
2016-12-07_01:42:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 178_of_200
2016-12-07_01:42:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 179_of_200
2016-12-07_01:42:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 179_of_200
2016-12-07_01:42:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 17_of_200
2016-12-07_01:42:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 17_of_200
2016-12-07_01:42:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 180_of_200
2016-12-07_01:42:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 180_of_200
2016-12-07_01:42:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 181_of_200
2016-12-07_01:42:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 181_of_200
2016-12-07_01:42:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 182_of_200
2016-12-07_01:42:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 182_of_200
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 183_of_200
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 183_of_200
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 184_of_200
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 184_of_200
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 185_of_200
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:42:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 185_of_200
2016-12-07_01:42:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 231358464 rss
2016-12-07_01:42:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 16.0500 user 0.9700 sys 32 cpus -- interval 10.0000 seconds 3.1400 user 0.1500 sys
2016-12-07_01:42:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 2254608 tx 7044610 rx -- interval 10.0001 seconds 568775 tx 960186 rx
2016-12-07_01:42:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 186_of_200
2016-12-07_01:42:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 186_of_200
2016-12-07_01:42:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 187_of_200
2016-12-07_01:42:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 187_of_200
2016-12-07_01:42:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 188_of_200
2016-12-07_01:42:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 188_of_200
2016-12-07_01:42:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 189_of_200
2016-12-07_01:42:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 189_of_200
2016-12-07_01:42:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 18_of_200
2016-12-07_01:42:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 18_of_200
2016-12-07_01:42:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 190_of_200
2016-12-07_01:42:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 190_of_200
2016-12-07_01:42:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 191_of_200
2016-12-07_01:42:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 191_of_200
2016-12-07_01:42:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 192_of_200
2016-12-07_01:42:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 192_of_200
2016-12-07_01:42:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 193_of_200
2016-12-07_01:42:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 193_of_200
2016-12-07_01:42:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 194_of_200
2016-12-07_01:42:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 194_of_200
2016-12-07_01:42:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 195_of_200
2016-12-07_01:42:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 195_of_200
2016-12-07_01:42:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 196_of_200
2016-12-07_01:42:57 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 196_of_200
2016-12-07_01:42:57 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 197_of_200
2016-12-07_01:42:57 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 197_of_200
2016-12-07_01:42:57 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 198_of_200
2016-12-07_01:42:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 198_of_200
2016-12-07_01:42:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 199_of_200
2016-12-07_01:42:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 199_of_200
2016-12-07_01:42:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 19_of_200
2016-12-07_01:42:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 19_of_200
2016-12-07_01:42:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 1_of_200
2016-12-07_01:42:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 1_of_200
2016-12-07_01:42:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 200_of_200
2016-12-07_01:42:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 200_of_200
2016-12-07_01:42:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 20_of_200
2016-12-07_01:43:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 20_of_200
2016-12-07_01:43:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 21_of_200
2016-12-07_01:43:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 21_of_200
2016-12-07_01:43:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 22_of_200
2016-12-07_01:43:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 22_of_200
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 23_of_200
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 23_of_200
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 24_of_200
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 24_of_200
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 25_of_200
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:43:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 25_of_200
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 26_of_200
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 260698112 rss
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 19.2800 user 1.1400 sys 32 cpus -- interval 10.0000 seconds 3.2300 user 0.1700 sys
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 2823093 tx 8011425 rx -- interval 10.0000 seconds 568485 tx 966815 rx
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 26_of_200
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 27_of_200
2016-12-07_01:43:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 27_of_200
2016-12-07_01:43:03 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 28_of_200
2016-12-07_01:43:03 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 28_of_200
2016-12-07_01:43:03 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 29_of_200
2016-12-07_01:43:03 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 29_of_200
2016-12-07_01:43:03 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 2_of_200
2016-12-07_01:43:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 2_of_200
2016-12-07_01:43:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 30_of_200
2016-12-07_01:43:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 30_of_200
2016-12-07_01:43:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 31_of_200
2016-12-07_01:43:05 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 31_of_200
2016-12-07_01:43:05 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 32_of_200
2016-12-07_01:43:05 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 32_of_200
2016-12-07_01:43:05 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 33_of_200
2016-12-07_01:43:05 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 33_of_200
2016-12-07_01:43:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 34_of_200
2016-12-07_01:43:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 34_of_200
2016-12-07_01:43:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 35_of_200
2016-12-07_01:43:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 35_of_200
2016-12-07_01:43:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 36_of_200
2016-12-07_01:43:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 36_of_200
2016-12-07_01:43:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 37_of_200
2016-12-07_01:43:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 37_of_200
2016-12-07_01:43:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 38_of_200
2016-12-07_01:43:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 38_of_200
2016-12-07_01:43:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 39_of_200
2016-12-07_01:43:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 39_of_200
2016-12-07_01:43:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 3_of_200
2016-12-07_01:43:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 3_of_200
2016-12-07_01:43:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 40_of_200
2016-12-07_01:43:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 40_of_200
2016-12-07_01:43:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 41_of_200
2016-12-07_01:43:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 41_of_200
2016-12-07_01:43:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 42_of_200
2016-12-07_01:43:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 42_of_200
2016-12-07_01:43:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 43_of_200
2016-12-07_01:43:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 43_of_200
2016-12-07_01:43:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 44_of_200
2016-12-07_01:43:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 44_of_200
2016-12-07_01:43:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 45_of_200
2016-12-07_01:43:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 45_of_200
2016-12-07_01:43:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 46_of_200
2016-12-07_01:43:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 46_of_200
2016-12-07_01:43:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 47_of_200
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 47_of_200
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 289234944 rss
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 22.4300 user 1.3200 sys 32 cpus -- interval 10.0000 seconds 3.1500 user 0.1800 sys
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 3389865 tx 8969660 rx -- interval 10.0000 seconds 566772 tx 958235 rx
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 48_of_200
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 48_of_200
2016-12-07_01:43:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 49_of_200
2016-12-07_01:43:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 49_of_200
2016-12-07_01:43:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 4_of_200
2016-12-07_01:43:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 4_of_200
2016-12-07_01:43:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 50_of_200
2016-12-07_01:43:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 50_of_200
2016-12-07_01:43:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 51_of_200
2016-12-07_01:43:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 51_of_200
2016-12-07_01:43:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 52_of_200
2016-12-07_01:43:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 52_of_200
2016-12-07_01:43:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 53_of_200
2016-12-07_01:43:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 53_of_200
2016-12-07_01:43:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 54_of_200
2016-12-07_01:43:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 54_of_200
2016-12-07_01:43:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 55_of_200
2016-12-07_01:43:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 55_of_200
2016-12-07_01:43:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 56_of_200
2016-12-07_01:43:16 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 56_of_200
2016-12-07_01:43:16 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 57_of_200
2016-12-07_01:43:16 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 57_of_200
2016-12-07_01:43:16 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 58_of_200
2016-12-07_01:43:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 58_of_200
2016-12-07_01:43:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 59_of_200
2016-12-07_01:43:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 59_of_200
2016-12-07_01:43:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 5_of_200
2016-12-07_01:43:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 5_of_200
2016-12-07_01:43:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 60_of_200
2016-12-07_01:43:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 60_of_200
2016-12-07_01:43:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 61_of_200
2016-12-07_01:43:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 61_of_200
2016-12-07_01:43:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 62_of_200
2016-12-07_01:43:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 62_of_200
2016-12-07_01:43:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 63_of_200
2016-12-07_01:43:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 63_of_200
2016-12-07_01:43:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 64_of_200
2016-12-07_01:43:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 64_of_200
2016-12-07_01:43:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 65_of_200
2016-12-07_01:43:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 65_of_200
2016-12-07_01:43:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 66_of_200
2016-12-07_01:43:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 66_of_200
2016-12-07_01:43:20 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 67_of_200
2016-12-07_01:43:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 67_of_200
2016-12-07_01:43:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 68_of_200
2016-12-07_01:43:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 68_of_200
2016-12-07_01:43:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 69_of_200
2016-12-07_01:43:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 69_of_200
2016-12-07_01:43:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:43:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 6_of_200
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 316506112 rss
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 25.7600 user 1.4800 sys 32 cpus -- interval 10.0000 seconds 3.3300 user 0.1600 sys
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 3961030 tx 9948690 rx -- interval 10.0000 seconds 571165 tx 979030 rx
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 6_of_200
2016-12-07_01:43:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 70_of_200
2016-12-07_01:43:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 70_of_200
2016-12-07_01:43:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 71_of_200
2016-12-07_01:43:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 71_of_200
2016-12-07_01:43:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 72_of_200
2016-12-07_01:43:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 72_of_200
2016-12-07_01:43:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 73_of_200
2016-12-07_01:43:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 73_of_200
2016-12-07_01:43:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 74_of_200
2016-12-07_01:43:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 74_of_200
2016-12-07_01:43:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 75_of_200
2016-12-07_01:43:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 75_of_200
2016-12-07_01:43:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 76_of_200
2016-12-07_01:43:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 76_of_200
2016-12-07_01:43:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 77_of_200
2016-12-07_01:43:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 77_of_200
2016-12-07_01:43:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 78_of_200
2016-12-07_01:43:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 78_of_200
2016-12-07_01:43:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 79_of_200
2016-12-07_01:43:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 79_of_200
2016-12-07_01:43:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 7_of_200
2016-12-07_01:43:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 7_of_200
2016-12-07_01:43:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 80_of_200
2016-12-07_01:43:27 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 80_of_200
2016-12-07_01:43:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 81_of_200
2016-12-07_01:43:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 81_of_200
2016-12-07_01:43:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 82_of_200
2016-12-07_01:43:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 82_of_200
2016-12-07_01:43:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 83_of_200
2016-12-07_01:43:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 83_of_200
2016-12-07_01:43:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 84_of_200
2016-12-07_01:43:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 84_of_200
2016-12-07_01:43:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 85_of_200
2016-12-07_01:43:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 85_of_200
2016-12-07_01:43:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 86_of_200
2016-12-07_01:43:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 86_of_200
2016-12-07_01:43:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 87_of_200
2016-12-07_01:43:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 87_of_200
2016-12-07_01:43:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 88_of_200
2016-12-07_01:43:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 88_of_200
2016-12-07_01:43:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 89_of_200
2016-12-07_01:43:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 89_of_200
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 8_of_200
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 308916224 rss
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: cpu 29.1800 user 1.7000 sys 32 cpus -- interval 10.0000 seconds 3.4200 user 0.2200 sys
2016-12-07_01:43:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr crunchstat: net:eth0 4519011 tx 10846945 rx -- interval 10.0000 seconds 557981 tx 898255 rx
2016-12-07_01:43:35 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 8_of_200
2016-12-07_01:43:35 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 90_of_200
2016-12-07_01:43:35 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 90_of_200
2016-12-07_01:43:35 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 91_of_200
2016-12-07_01:43:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 91_of_200
2016-12-07_01:43:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 92_of_200
2016-12-07_01:43:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 92_of_200
2016-12-07_01:43:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 93_of_200
2016-12-07_01:43:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 93_of_200
2016-12-07_01:43:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 94_of_200
2016-12-07_01:43:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 94_of_200
2016-12-07_01:43:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 95_of_200
2016-12-07_01:43:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 95_of_200
2016-12-07_01:43:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 96_of_200
2016-12-07_01:43:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 96_of_200
2016-12-07_01:43:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 97_of_200
2016-12-07_01:43:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 97_of_200
2016-12-07_01:43:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 98_of_200
2016-12-07_01:43:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 98_of_200
2016-12-07_01:43:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 99_of_200
2016-12-07_01:43:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 99_of_200
2016-12-07_01:43:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Have 42 gVCFs in group 9_of_200
2016-12-07_01:43:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Creating task to process 9_of_200
2016-12-07_01:43:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 stderr Ending task 0 successfully
2016-12-07_01:43:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 child 32476 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:43:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 success in 99 seconds
2016-12-07_01:43:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 0 task output (0 bytes): 
2016-12-07_01:43:40 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 200 new tasks from API server.
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 1 done, 0 running, 200 todo
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237  wait for last 0 children to finish
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237  start level 1 with 8 slots
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 1 done, 0 running, 200 todo
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 job_task z8ta6-ot0gb-8m8fzgl40jp3gpq
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 child 35728 started on humgen-05-15.1
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 job_task z8ta6-ot0gb-ddfiivm4j19eemu
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 child 35738 started on humgen-05-15.2
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 job_task z8ta6-ot0gb-0ue98t5j0ynkmul
2016-12-07_01:43:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 child 35742 started on humgen-05-15.3
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 job_task z8ta6-ot0gb-lhhjff9kfdln0yy
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 child 35755 started on humgen-05-15.4
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 job_task z8ta6-ot0gb-w09ptjvzffwr5oi
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 child 35762 started on humgen-05-15.5
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 job_task z8ta6-ot0gb-bj1oekicgfkuolk
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 child 35783 started on humgen-05-15.6
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 job_task z8ta6-ot0gb-zvab8yxhku2d1r6
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 child 35792 started on humgen-05-15.7
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 job_task z8ta6-ot0gb-pyzh6vubwj7yt6r
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 child 35800 started on humgen-05-15.8
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.1.35728','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8m8fzgl40jp3gpq-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8m8fzgl40jp3gpq-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8m8fzgl40jp3gpq-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=1 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-8m8fzgl40jp3gpq --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8m8fzgl40jp3gpq-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8m8fzgl40jp3gpq-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=1 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-8m8fzgl40jp3gpq --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/06da3adb3b80783bd3cbd0898a6dff26afdded678f908f3eb419137303175679/cgroup.procs
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/06da3adb3b80783bd3cbd0898a6dff26afdded678f908f3eb419137303175679/memory.stat
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5066752 rss
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/06da3adb3b80783bd3cbd0898a6dff26afdded678f908f3eb419137303175679/cpuacct.stat
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/06da3adb3b80783bd3cbd0898a6dff26afdded678f908f3eb419137303175679/cpuset.cpus
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/06da3adb3b80783bd3cbd0898a6dff26afdded678f908f3eb419137303175679/blkio.io_service_bytes
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.2.35738','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ddfiivm4j19eemu-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ddfiivm4j19eemu-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ddfiivm4j19eemu-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=2 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ddfiivm4j19eemu --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.3.35742','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0ue98t5j0ynkmul-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0ue98t5j0ynkmul-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0ue98t5j0ynkmul-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=3 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-0ue98t5j0ynkmul --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.4.35755','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-lhhjff9kfdln0yy-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-lhhjff9kfdln0yy-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-lhhjff9kfdln0yy-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=4 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-lhhjff9kfdln0yy --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.5.35762','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w09ptjvzffwr5oi-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-w09ptjvzffwr5oi-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w09ptjvzffwr5oi-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=5 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-w09ptjvzffwr5oi --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.6.35783','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-bj1oekicgfkuolk-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-bj1oekicgfkuolk-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-bj1oekicgfkuolk-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=6 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-bj1oekicgfkuolk --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.7.35792','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zvab8yxhku2d1r6-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zvab8yxhku2d1r6-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zvab8yxhku2d1r6-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=7 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-zvab8yxhku2d1r6 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.8.35800','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-pyzh6vubwj7yt6r-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-pyzh6vubwj7yt6r-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-pyzh6vubwj7yt6r-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=8 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-pyzh6vubwj7yt6r --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:43:51 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 1 done, 8 running, 192 todo
2016-12-07_01:43:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Mounting task input collection to get interval_list
2016-12-07_01:43:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Interval_List collection mounted at /keep/812316f6c95d8641412912523842bbfc+94
2016-12-07_01:43:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Total chunk length is 15618872
2016-12-07_01:43:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:43:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Have 1 intervals
2016-12-07_01:43:52 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:43:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:43:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:43:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:43:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Have 5400 potential reusable task outputs
2016-12-07_01:43:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Have 5400 potentially reusable tasks
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Creating task to process interval: [9:6148617-21767488]
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr No reusable JobTask matched key parameters [u'100_of_200', u'a241297995deeab501f9bd99dc533315+14700', '9:6148617-21767488', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 stderr Ending task 1 successfully
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 child 35728 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 success in 10 seconds
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237 1 task output (0 bytes): 
2016-12-07_01:44:00 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 2 done, 7 running, 193 todo
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 job_task z8ta6-ot0gb-jwhcld7a4617p0u
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 child 36333 started on humgen-05-15.1
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.9.36333','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jwhcld7a4617p0u-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-jwhcld7a4617p0u-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jwhcld7a4617p0u-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=9 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-jwhcld7a4617p0u --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ddfiivm4j19eemu-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ddfiivm4j19eemu-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=2 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ddfiivm4j19eemu --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/750c846006983a718c2d2de06c7e523eb68f523de48aa105eee010d8de9b360b/cgroup.procs
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/750c846006983a718c2d2de06c7e523eb68f523de48aa105eee010d8de9b360b/memory.stat
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 552960 rss
2016-12-07_01:44:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/750c846006983a718c2d2de06c7e523eb68f523de48aa105eee010d8de9b360b/cpuacct.stat
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/750c846006983a718c2d2de06c7e523eb68f523de48aa105eee010d8de9b360b/cpuset.cpus
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: cpu 0.0100 user 0.0000 sys 32 cpus
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/750c846006983a718c2d2de06c7e523eb68f523de48aa105eee010d8de9b360b/blkio.io_service_bytes
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Mounting task input collection to get interval_list
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Interval_List collection mounted at /keep/6af141a235ccb024f6a32fc12ee98dcf+94
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Total chunk length is 15618872
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Have 1 intervals
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:44:08 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 2 done, 8 running, 192 todo
2016-12-07_01:44:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:44:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:44:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:44:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:44:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:44:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:44:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: cpu 0.4800 user 0.1800 sys 32 cpus -- interval 10.0000 seconds 0.4700 user 0.1800 sys
2016-12-07_01:44:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr crunchstat: net:eth0 21811 tx 237951 rx
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Have 5400 potential reusable task outputs
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Have 5400 potentially reusable tasks
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Creating task to process interval: [9:21767489-37386360]
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr No reusable JobTask matched key parameters [u'101_of_200', u'1b848222e216cbae87e6932d5820a285+14700', '9:21767489-37386360', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 stderr Ending task 1 successfully
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 child 35738 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 success in 22 seconds
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 2 task output (0 bytes): 
2016-12-07_01:44:12 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 3 done, 7 running, 193 todo
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 job_task z8ta6-ot0gb-h2i93nvxu7c9ugg
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 child 36683 started on humgen-05-15.2
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.10.36683','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h2i93nvxu7c9ugg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-h2i93nvxu7c9ugg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h2i93nvxu7c9ugg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=10 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-h2i93nvxu7c9ugg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr srun: Job step created
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0ue98t5j0ynkmul-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0ue98t5j0ynkmul-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=3 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-0ue98t5j0ynkmul --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05/cgroup.procs
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05/memory.stat
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5025792 rss
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05/cpuacct.stat
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05/cpuset.cpus
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: cpu 0.0200 user 0.0100 sys 32 cpus
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05/blkio.io_service_bytes
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Mounting task input collection to get interval_list
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Interval_List collection mounted at /keep/2e7cf8719961c4cc5b2ffdd92ff99dd0+94
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Total chunk length is 15618872
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Have 1 intervals
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:44:19 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:44:20 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 3 done, 8 running, 192 todo
2016-12-07_01:44:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Have 5400 potential reusable task outputs
2016-12-07_01:44:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Have 5400 potentially reusable tasks
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Creating task to process interval: [9:37386361-53005232]
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr No reusable JobTask matched key parameters [u'102_of_200', u'0b01f4aa638023c7844e14aa840fe4cc+14532', '9:37386361-53005232', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr Ending task 1 successfully
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid 786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05, parent docker, root /sys/fs/cgroup
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid 786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05, parent docker, root /sys/fs/cgroup
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid 786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05, parent docker, root /sys/fs/cgroup
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid 786bbb1f7bce7c45e940006b8f291d55cb2bf89f93c5e93a5f05650ca5d6ed05, parent docker, root /sys/fs/cgroup
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr srun: Job step created
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 child 35742 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 success in 33 seconds
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237 3 task output (0 bytes): 
2016-12-07_01:44:23 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 4 done, 7 running, 193 todo
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 job_task z8ta6-ot0gb-emaap926s9c1lgu
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 child 37055 started on humgen-05-15.3
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.11.37055','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-emaap926s9c1lgu-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-emaap926s9c1lgu-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-emaap926s9c1lgu-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=11 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-emaap926s9c1lgu --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-bj1oekicgfkuolk-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-bj1oekicgfkuolk-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=6 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-bj1oekicgfkuolk --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/b66a219cdae739188b689dfa91946b3b49b12a94a760fb7810eae6ef092d7df3/cgroup.procs
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/b66a219cdae739188b689dfa91946b3b49b12a94a760fb7810eae6ef092d7df3/memory.stat
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4935680 rss
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/b66a219cdae739188b689dfa91946b3b49b12a94a760fb7810eae6ef092d7df3/cpuacct.stat
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/b66a219cdae739188b689dfa91946b3b49b12a94a760fb7810eae6ef092d7df3/cpuset.cpus
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/b66a219cdae739188b689dfa91946b3b49b12a94a760fb7810eae6ef092d7df3/blkio.io_service_bytes
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Mounting task input collection to get interval_list
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Interval_List collection mounted at /keep/4325c7dac5c39f2db578e849695a12aa+94
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Total chunk length is 15618872
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Have 1 intervals
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:44:30 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:44:31 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 4 done, 8 running, 192 todo
2016-12-07_01:44:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Have 5400 potential reusable task outputs
2016-12-07_01:44:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:44:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:44:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Have 5400 potentially reusable tasks
2016-12-07_01:44:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Creating task to process interval: [9:84242977-99861848]
2016-12-07_01:44:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr No reusable JobTask matched key parameters [u'105_of_200', u'3a3bb24e84ba30a06b3d46caa3b1c040+14700', '9:84242977-99861848', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:44:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr Ending task 1 successfully
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 child 35783 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 success in 43 seconds
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 6 task output (0 bytes): 
2016-12-07_01:44:34 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:44:40 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 5 done, 7 running, 193 todo
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 job_task z8ta6-ot0gb-emp7b3s0czu568e
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 child 37378 started on humgen-05-15.6
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr srun: Job step created
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-jwhcld7a4617p0u-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jwhcld7a4617p0u-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=9 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-jwhcld7a4617p0u --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d26b921a453a3747a9ccc6a8dccefb1ece58d4c6faea636c7b5f26e08b1bb501/cgroup.procs
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d26b921a453a3747a9ccc6a8dccefb1ece58d4c6faea636c7b5f26e08b1bb501/memory.stat
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4317184 rss
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d26b921a453a3747a9ccc6a8dccefb1ece58d4c6faea636c7b5f26e08b1bb501/cpuacct.stat
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d26b921a453a3747a9ccc6a8dccefb1ece58d4c6faea636c7b5f26e08b1bb501/cpuset.cpus
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d26b921a453a3747a9ccc6a8dccefb1ece58d4c6faea636c7b5f26e08b1bb501/blkio.io_service_bytes
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Mounting task input collection to get interval_list
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Interval_List collection mounted at /keep/64696f9b403b7674c728386882cbd3b7+94
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Total chunk length is 15498872
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Have 1 intervals
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.12.37378','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-emp7b3s0czu568e-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-emp7b3s0czu568e-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-emp7b3s0czu568e-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=12 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-emp7b3s0czu568e --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:44:41 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 5 done, 8 running, 192 todo
2016-12-07_01:44:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:44:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:44:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:44:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:44:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: blkio:0:0 0 write 12564 read -- interval 10.0000 seconds 0 write 12564 read
2016-12-07_01:44:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:44:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: cpu 0.4900 user 0.1800 sys 32 cpus -- interval 10.0001 seconds 0.4500 user 0.1600 sys
2016-12-07_01:44:44 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr crunchstat: net:eth0 21877 tx 238231 rx
2016-12-07_01:44:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Have 5400 potential reusable task outputs
2016-12-07_01:44:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:44:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:44:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Have 5400 potentially reusable tasks
2016-12-07_01:44:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Creating task to process interval: [9:131099593-141213431 10:1-5385033]
2016-12-07_01:44:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr No reusable JobTask matched key parameters [u'108_of_200', u'7e64dee294f55ab4bfb56c1a154618f8+14704', '9:131099593-141213431 10:1-5385033', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:44:46 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 stderr Ending task 1 successfully
2016-12-07_01:44:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 child 36333 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:44:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 success in 40 seconds
2016-12-07_01:44:47 z8ta6-8i9sb-r52unvxj6avgrio 32237 9 task output (0 bytes): 
2016-12-07_01:44:47 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:44:53 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:44:53 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 6 done, 7 running, 193 todo
2016-12-07_01:44:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 job_task z8ta6-ot0gb-h1dvg3wbjrftvyc
2016-12-07_01:44:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 child 37745 started on humgen-05-15.1
2016-12-07_01:44:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.13.37745','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h1dvg3wbjrftvyc-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-h1dvg3wbjrftvyc-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h1dvg3wbjrftvyc-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=13 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-h1dvg3wbjrftvyc --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:44:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr srun: Job step created
2016-12-07_01:44:53 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-w09ptjvzffwr5oi-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w09ptjvzffwr5oi-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=5 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-w09ptjvzffwr5oi --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/160ed6210d32ce45e3aaa57a46c3c33629fc4129a79d3b17b0bdfcab05589676/cgroup.procs
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/160ed6210d32ce45e3aaa57a46c3c33629fc4129a79d3b17b0bdfcab05589676/memory.stat
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5996544 rss
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/160ed6210d32ce45e3aaa57a46c3c33629fc4129a79d3b17b0bdfcab05589676/cpuacct.stat
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/160ed6210d32ce45e3aaa57a46c3c33629fc4129a79d3b17b0bdfcab05589676/cpuset.cpus
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/160ed6210d32ce45e3aaa57a46c3c33629fc4129a79d3b17b0bdfcab05589676/blkio.io_service_bytes
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Mounting task input collection to get interval_list
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Interval_List collection mounted at /keep/61144aa3079bc748eb474251bb12d2af+94
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Total chunk length is 15618872
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Have 1 intervals
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:44:54 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 6 done, 8 running, 192 todo
2016-12-07_01:44:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 notice: task is not in slurm queue but srun process 35800 has not exited
2016-12-07_01:44:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 notice: task is not in slurm queue but srun process 35755 has not exited
2016-12-07_01:44:55 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 notice: task is not in slurm queue but srun process 35792 has not exited
2016-12-07_01:44:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Have 5400 potential reusable task outputs
2016-12-07_01:44:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:44:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:44:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Have 5400 potentially reusable tasks
2016-12-07_01:44:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Creating task to process interval: [9:68624105-84242976]
2016-12-07_01:44:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr No reusable JobTask matched key parameters [u'104_of_200', u'0e7a8a6d12ed4d9a7f7a47c972f98c08+14700', '9:68624105-84242976', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:44:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 stderr Ending task 1 successfully
2016-12-07_01:44:57 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 child 35762 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:44:57 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 success in 66 seconds
2016-12-07_01:44:57 z8ta6-8i9sb-r52unvxj6avgrio 32237 5 task output (0 bytes): 
2016-12-07_01:44:57 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 7 done, 7 running, 193 todo
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 job_task z8ta6-ot0gb-44rzd531e42fbx8
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 child 38131 started on humgen-05-15.5
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.14.38131','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-44rzd531e42fbx8-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-44rzd531e42fbx8-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-44rzd531e42fbx8-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=14 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-44rzd531e42fbx8 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr srun: Job step created
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-emp7b3s0czu568e-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-emp7b3s0czu568e-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=12 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-emp7b3s0czu568e --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/65ec2471dcc6fd71883704a4cd70cd6d5c532d459819ea431d1ce1e623ec3e25/cgroup.procs
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/65ec2471dcc6fd71883704a4cd70cd6d5c532d459819ea431d1ce1e623ec3e25/memory.stat
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 3129344 rss
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/65ec2471dcc6fd71883704a4cd70cd6d5c532d459819ea431d1ce1e623ec3e25/cpuacct.stat
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/65ec2471dcc6fd71883704a4cd70cd6d5c532d459819ea431d1ce1e623ec3e25/cpuset.cpus
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: cpu 0.0100 user 0.0000 sys 32 cpus
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/65ec2471dcc6fd71883704a4cd70cd6d5c532d459819ea431d1ce1e623ec3e25/blkio.io_service_bytes
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Mounting task input collection to get interval_list
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Interval_List collection mounted at /keep/11837032d32bcfff7055a484f9c0ddb8+94
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Total chunk length is 15618872
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Have 1 intervals
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:45:04 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 7 done, 8 running, 192 todo
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Have 5400 potential reusable task outputs
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Have 5400 potentially reusable tasks
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Creating task to process interval: [10:21003906-36622777]
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr No reusable JobTask matched key parameters [u'110_of_200', u'e20f5f00b25cd7e53f799770dbe9e308+14700', '10:21003906-36622777', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr Ending task 1 successfully
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr srun: Job step created
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 child 37378 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:45:07 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 success in 26 seconds
2016-12-07_01:45:08 z8ta6-8i9sb-r52unvxj6avgrio 32237 12 task output (0 bytes): 
2016-12-07_01:45:08 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 8 done, 7 running, 193 todo
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 job_task z8ta6-ot0gb-bsoepr8ldxtb066
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 child 38743 started on humgen-05-15.6
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-h2i93nvxu7c9ugg-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h2i93nvxu7c9ugg-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=10 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-h2i93nvxu7c9ugg --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ebbc87bbb5f70a58992f217792e1637234a7c5ee751d3811388871f18de09754/cgroup.procs
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/ebbc87bbb5f70a58992f217792e1637234a7c5ee751d3811388871f18de09754/memory.stat
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4939776 rss
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ebbc87bbb5f70a58992f217792e1637234a7c5ee751d3811388871f18de09754/cpuacct.stat
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/ebbc87bbb5f70a58992f217792e1637234a7c5ee751d3811388871f18de09754/cpuset.cpus
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/ebbc87bbb5f70a58992f217792e1637234a7c5ee751d3811388871f18de09754/blkio.io_service_bytes
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Mounting task input collection to get interval_list
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Interval_List collection mounted at /keep/8f01b9c543342e361e6b7738dc97925d+94
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Total chunk length is 15618872
2016-12-07_01:45:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:45:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Have 1 intervals
2016-12-07_01:45:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:45:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:45:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:45:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:45:15 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.15.38743','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-bsoepr8ldxtb066-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-bsoepr8ldxtb066-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-bsoepr8ldxtb066-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=15 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-bsoepr8ldxtb066 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:15 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 8 done, 8 running, 192 todo
2016-12-07_01:45:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Have 5400 potential reusable task outputs
2016-12-07_01:45:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:45:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:45:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Have 5400 potentially reusable tasks
2016-12-07_01:45:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Creating task to process interval: [10:5385034-21003905]
2016-12-07_01:45:17 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr No reusable JobTask matched key parameters [u'109_of_200', u'b8ca1138c80f500be618798294067825+14700', '10:5385034-21003905', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr Ending task 1 successfully
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 child 36683 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 success in 59 seconds
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 10 task output (0 bytes): 
2016-12-07_01:45:18 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 9 done, 7 running, 193 todo
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 job_task z8ta6-ot0gb-obwk7r1dxx28jru
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 child 39067 started on humgen-05-15.2
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.16.39067','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-obwk7r1dxx28jru-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-obwk7r1dxx28jru-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-obwk7r1dxx28jru-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=16 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-obwk7r1dxx28jru --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr srun: Job step created
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-44rzd531e42fbx8-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-44rzd531e42fbx8-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=14 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-44rzd531e42fbx8 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/99b938ed7ed8bc1f7adcd7753dc39a32b6617185dd176f6e9cb5d248615d6c1d/cgroup.procs
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/99b938ed7ed8bc1f7adcd7753dc39a32b6617185dd176f6e9cb5d248615d6c1d/memory.stat
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5492736 rss
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/99b938ed7ed8bc1f7adcd7753dc39a32b6617185dd176f6e9cb5d248615d6c1d/cpuacct.stat
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/99b938ed7ed8bc1f7adcd7753dc39a32b6617185dd176f6e9cb5d248615d6c1d/cpuset.cpus
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/99b938ed7ed8bc1f7adcd7753dc39a32b6617185dd176f6e9cb5d248615d6c1d/blkio.io_service_bytes
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Mounting task input collection to get interval_list
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Interval_List collection mounted at /keep/fe3ded28e7bf89d25f8d4e08ecc0d6c5+94
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Total chunk length is 15618872
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Have 1 intervals
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:45:25 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 9 done, 8 running, 192 todo
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Have 5400 potential reusable task outputs
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Have 5400 potentially reusable tasks
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Creating task to process interval: [10:52241650-67860521]
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr No reusable JobTask matched key parameters [u'112_of_200', u'9e1996fed17b6a2fca279a862266ce9d+14700', '10:52241650-67860521', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr Ending task 1 successfully
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:45:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 killing orphaned srun process 35800 (task not in slurm queue, no stderr received in last 89s)
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 sending 2x signal 2 to pid 35800
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 killing orphaned srun process 35755 (task not in slurm queue, no stderr received in last 89s)
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 sending 2x signal 2 to pid 35755
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 killing orphaned srun process 35792 (task not in slurm queue, no stderr received in last 89s)
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 sending 2x signal 2 to pid 35792
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr srun: Job step created
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 child 35755 on humgen-05-15.4 exit 1 success=
2016-12-07_01:45:29 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:45:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 failure (#1, temporary) after 98 seconds
2016-12-07_01:45:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 4 task output (0 bytes): 
2016-12-07_01:45:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 child 35800 on humgen-05-15.8 exit 1 success=
2016-12-07_01:45:31 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:45:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 failure (#1, temporary) after 100 seconds
2016-12-07_01:45:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 8 task output (0 bytes): 
2016-12-07_01:45:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 child 38131 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:45:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 success in 29 seconds
2016-12-07_01:45:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 14 task output (0 bytes): 
2016-12-07_01:45:33 z8ta6-8i9sb-r52unvxj6avgrio 32237  3 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:45:39 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:45:39 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 10 done, 5 running, 195 todo
2016-12-07_01:45:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 job_task z8ta6-ot0gb-f86lygj6mewa7a6
2016-12-07_01:45:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 child 39539 started on humgen-05-15.4
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 job_task z8ta6-ot0gb-g3qgtrm3ymgzwry
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 child 39552 started on humgen-05-15.8
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 job_task z8ta6-ot0gb-2uxsl19t6f1g869
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 child 39557 started on humgen-05-15.5
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr srun: Job step created
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-obwk7r1dxx28jru-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-obwk7r1dxx28jru-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=16 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-obwk7r1dxx28jru --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-emaap926s9c1lgu-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-emaap926s9c1lgu-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=11 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-emaap926s9c1lgu --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/94e4ca43c7a30f0bcd5c46a43f17f3147fd7a5ebd9ac2769216fc81969d81a33/cgroup.procs
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/94e4ca43c7a30f0bcd5c46a43f17f3147fd7a5ebd9ac2769216fc81969d81a33/memory.stat
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4780032 rss
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/94e4ca43c7a30f0bcd5c46a43f17f3147fd7a5ebd9ac2769216fc81969d81a33/cpuacct.stat
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/94e4ca43c7a30f0bcd5c46a43f17f3147fd7a5ebd9ac2769216fc81969d81a33/cpuset.cpus
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/94e4ca43c7a30f0bcd5c46a43f17f3147fd7a5ebd9ac2769216fc81969d81a33/blkio.io_service_bytes
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Mounting task input collection to get interval_list
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Interval_List collection mounted at /keep/47924fc036d6a931b5f0cf53440e0969+92
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Total chunk length is 15618872
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Have 1 intervals
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Have 5400 potential reusable task outputs
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Have 5400 potentially reusable tasks
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Creating task to process interval: [1:140569849-156188720]
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr No reusable JobTask matched key parameters [u'10_of_200', u'a99a25ff7de535672c0817451b92bd1a+14616', '1:140569849-156188720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr Ending task 1 successfully
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.17.39539','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-f86lygj6mewa7a6-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-f86lygj6mewa7a6-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-f86lygj6mewa7a6-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=17 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-f86lygj6mewa7a6 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.18.39552','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-g3qgtrm3ymgzwry-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-g3qgtrm3ymgzwry-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-g3qgtrm3ymgzwry-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=18 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-g3qgtrm3ymgzwry --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.19.39557','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2uxsl19t6f1g869-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2uxsl19t6f1g869-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2uxsl19t6f1g869-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=19 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2uxsl19t6f1g869 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 child 35792 on humgen-05-15.7 exit 1 success=
2016-12-07_01:45:40 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:45:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 failure (#1, temporary) after 109 seconds
2016-12-07_01:45:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 7 task output (0 bytes): 
2016-12-07_01:45:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 child 37055 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:45:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 success in 72 seconds
2016-12-07_01:45:42 z8ta6-8i9sb-r52unvxj6avgrio 32237 11 task output (0 bytes): 
2016-12-07_01:45:42 z8ta6-8i9sb-r52unvxj6avgrio 32237  2 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 11 done, 6 running, 194 todo
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 job_task z8ta6-ot0gb-rdeq39q4vvj65qo
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 child 39805 started on humgen-05-15.7
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 job_task z8ta6-ot0gb-aa2zvm6r9qg4l32
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 child 39814 started on humgen-05-15.3
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f885e53a1b71f333221072cd967fe37f81ea4cb54280a67c06c5171f3ff94105/cgroup.procs
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/f885e53a1b71f333221072cd967fe37f81ea4cb54280a67c06c5171f3ff94105/memory.stat
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7733248 rss
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f885e53a1b71f333221072cd967fe37f81ea4cb54280a67c06c5171f3ff94105/cpuacct.stat
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/f885e53a1b71f333221072cd967fe37f81ea4cb54280a67c06c5171f3ff94105/cpuset.cpus
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/f885e53a1b71f333221072cd967fe37f81ea4cb54280a67c06c5171f3ff94105/blkio.io_service_bytes
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Mounting task input collection to get interval_list
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Interval_List collection mounted at /keep/8b0a6de6be0a7b3dd7360336477e02fd+94
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Total chunk length is 15618872
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Have 1 intervals
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.20.39805','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-rdeq39q4vvj65qo-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-rdeq39q4vvj65qo-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-rdeq39q4vvj65qo-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=20 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-rdeq39q4vvj65qo --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.21.39814','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-aa2zvm6r9qg4l32-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-aa2zvm6r9qg4l32-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-aa2zvm6r9qg4l32-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=21 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-aa2zvm6r9qg4l32 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 11 done, 8 running, 192 todo
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Have 5400 potential reusable task outputs
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Have 5400 potentially reusable tasks
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Creating task to process interval: [10:83479394-99098265]
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr No reusable JobTask matched key parameters [u'114_of_200', u'c3bc3b07edcf5d2b7b9bfb2b5154d56f+14700', '10:83479394-99098265', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 49078272 rss
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: cpu 0.7600 user 0.1600 sys 32 cpus -- interval 10.0001 seconds 0.7200 user 0.1500 sys
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr crunchstat: net:eth0 37904 tx 3243470 rx
2016-12-07_01:45:50 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 stderr Ending task 1 successfully
2016-12-07_01:45:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 child 39067 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:45:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 success in 26 seconds
2016-12-07_01:45:51 z8ta6-8i9sb-r52unvxj6avgrio 32237 16 task output (0 bytes): 
2016-12-07_01:45:51 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:45:57 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 12 done, 7 running, 193 todo
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 job_task z8ta6-ot0gb-g6whygrqqg4quk3
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 child 40085 started on humgen-05-15.2
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.22.40085','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-g6whygrqqg4quk3-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-g6whygrqqg4quk3-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-g6whygrqqg4quk3-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=22 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-g6whygrqqg4quk3 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-f86lygj6mewa7a6-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-f86lygj6mewa7a6-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=17 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-f86lygj6mewa7a6 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/4da75b3788e0dd26b148779b90df9a64855cadb91305ed0306ed1ef8dd2b3e42/cgroup.procs
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/4da75b3788e0dd26b148779b90df9a64855cadb91305ed0306ed1ef8dd2b3e42/memory.stat
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4653056 rss
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/4da75b3788e0dd26b148779b90df9a64855cadb91305ed0306ed1ef8dd2b3e42/cpuacct.stat
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/4da75b3788e0dd26b148779b90df9a64855cadb91305ed0306ed1ef8dd2b3e42/cpuset.cpus
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/4da75b3788e0dd26b148779b90df9a64855cadb91305ed0306ed1ef8dd2b3e42/blkio.io_service_bytes
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Mounting task input collection to get interval_list
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Interval_List collection mounted at /keep/e05c9a56f87bbb0e87bd78edfb2138bb+94
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Total chunk length is 15618872
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Have 1 intervals
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:45:58 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 12 done, 8 running, 192 todo
2016-12-07_01:46:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:46:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:46:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:46:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:46:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: blkio:0:0 0 write 12527 read -- interval 10.0000 seconds 0 write 12527 read
2016-12-07_01:46:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:46:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: cpu 0.4800 user 0.1400 sys 32 cpus -- interval 10.0001 seconds 0.4500 user 0.1400 sys
2016-12-07_01:46:01 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr crunchstat: net:eth0 21877 tx 238348 rx
2016-12-07_01:46:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Have 5400 potential reusable task outputs
2016-12-07_01:46:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:46:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:46:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Have 5400 potentially reusable tasks
2016-12-07_01:46:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Creating task to process interval: [10:99098266-114717137]
2016-12-07_01:46:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr No reusable JobTask matched key parameters [u'115_of_200', u'5601c60d4022448c1a998597fc7d32dc+14700', '10:99098266-114717137', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:46:02 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 stderr Ending task 1 successfully
2016-12-07_01:46:03 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 child 39539 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:46:03 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 success in 24 seconds
2016-12-07_01:46:03 z8ta6-8i9sb-r52unvxj6avgrio 32237 17 task output (0 bytes): 
2016-12-07_01:46:03 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:46:09 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:46:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 notice: task is not in slurm queue but srun process 37745 has not exited
2016-12-07_01:46:09 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 13 done, 7 running, 193 todo
2016-12-07_01:46:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 job_task z8ta6-ot0gb-npaa8flwmec91iq
2016-12-07_01:46:09 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 child 40424 started on humgen-05-15.4
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.23.40424','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-npaa8flwmec91iq-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-npaa8flwmec91iq-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-npaa8flwmec91iq-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=23 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-npaa8flwmec91iq --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-aa2zvm6r9qg4l32-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-aa2zvm6r9qg4l32-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=21 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-aa2zvm6r9qg4l32 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/db52c487c58aa36460cea6dfe99a7727cd24aa3958069e1f5a900048b1b99315/cgroup.procs
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/db52c487c58aa36460cea6dfe99a7727cd24aa3958069e1f5a900048b1b99315/memory.stat
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5943296 rss
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/db52c487c58aa36460cea6dfe99a7727cd24aa3958069e1f5a900048b1b99315/cpuacct.stat
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/db52c487c58aa36460cea6dfe99a7727cd24aa3958069e1f5a900048b1b99315/cpuset.cpus
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/db52c487c58aa36460cea6dfe99a7727cd24aa3958069e1f5a900048b1b99315/blkio.io_service_bytes
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: net:eth0 0 tx 0 rx
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Mounting task input collection to get interval_list
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Interval_List collection mounted at /keep/dbba1077cec452fe00018827674a6082+94
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Total chunk length is 15618872
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Have 1 intervals
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:46:10 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 13 done, 8 running, 192 todo
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Have 5400 potential reusable task outputs
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 45322240 rss
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: cpu 0.6200 user 0.1900 sys 32 cpus -- interval 10.0002 seconds 0.5800 user 0.1900 sys
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr crunchstat: net:eth0 27463 tx 3234376 rx -- interval 10.0002 seconds 27463 tx 3234376 rx
2016-12-07_01:46:13 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Have 5400 potentially reusable tasks
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Creating task to process interval: [11:25919007-41537878]
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr No reusable JobTask matched key parameters [u'119_of_200', u'e715710b28adf5479c2b4ee6a270292a+14700', '11:25919007-41537878', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 stderr Ending task 1 successfully
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr srun: Job step created
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 child 39814 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 success in 25 seconds
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237 21 task output (0 bytes): 
2016-12-07_01:46:14 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 14 done, 7 running, 193 todo
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 job_task z8ta6-ot0gb-jzrtrwxfqp0jbgp
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 child 40806 started on humgen-05-15.3
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-rdeq39q4vvj65qo-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-rdeq39q4vvj65qo-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=20 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-rdeq39q4vvj65qo --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/078d10980f4d305f977e22701cdf31741e918d9f8d7ccc6d1fe170e703dad63a/cgroup.procs
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/078d10980f4d305f977e22701cdf31741e918d9f8d7ccc6d1fe170e703dad63a/memory.stat
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5435392 rss
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/078d10980f4d305f977e22701cdf31741e918d9f8d7ccc6d1fe170e703dad63a/cpuacct.stat
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/078d10980f4d305f977e22701cdf31741e918d9f8d7ccc6d1fe170e703dad63a/cpuset.cpus
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/078d10980f4d305f977e22701cdf31741e918d9f8d7ccc6d1fe170e703dad63a/blkio.io_service_bytes
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Mounting task input collection to get interval_list
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Interval_List collection mounted at /keep/d9aa702cd7b3a1e6b6dec54b4a3d8e35+94
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Total chunk length is 15618872
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Have 1 intervals
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:46:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.24.40806','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jzrtrwxfqp0jbgp-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-jzrtrwxfqp0jbgp-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jzrtrwxfqp0jbgp-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=24 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-jzrtrwxfqp0jbgp --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:46:22 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 14 done, 8 running, 192 todo
2016-12-07_01:46:24 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 notice: task is not in slurm queue but srun process 38743 has not exited
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24092672 rss
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: cpu 0.4900 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4500 user 0.1500 sys
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr crunchstat: net:eth0 21811 tx 238165 rx
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Have 5400 potential reusable task outputs
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Have 5400 potentially reusable tasks
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Creating task to process interval: [11:10300135-25919006]
2016-12-07_01:46:25 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr No reusable JobTask matched key parameters [u'118_of_200', u'79b22e2a5c2a107fd090c78ed23ff2e7+14700', '11:10300135-25919006', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:46:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 stderr Ending task 1 successfully
2016-12-07_01:46:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 child 39805 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:46:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 success in 37 seconds
2016-12-07_01:46:26 z8ta6-8i9sb-r52unvxj6avgrio 32237 20 task output (0 bytes): 
2016-12-07_01:46:26 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 15 done, 7 running, 193 todo
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 job_task z8ta6-ot0gb-w2chjt72sypegzz
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 child 41163 started on humgen-05-15.7
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.25.41163','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w2chjt72sypegzz-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-w2chjt72sypegzz-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w2chjt72sypegzz-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=25 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-w2chjt72sypegzz --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-npaa8flwmec91iq-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-npaa8flwmec91iq-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=23 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-npaa8flwmec91iq --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/c3ccf0392745602f6082c9ecd8d62f148af2796015d1765980a46be10956291a/cgroup.procs
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/c3ccf0392745602f6082c9ecd8d62f148af2796015d1765980a46be10956291a/memory.stat
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 602112 rss
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/c3ccf0392745602f6082c9ecd8d62f148af2796015d1765980a46be10956291a/cpuacct.stat
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/c3ccf0392745602f6082c9ecd8d62f148af2796015d1765980a46be10956291a/cpuset.cpus
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: cpu 0.0000 user 0.0000 sys 32 cpus
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/c3ccf0392745602f6082c9ecd8d62f148af2796015d1765980a46be10956291a/blkio.io_service_bytes
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Mounting task input collection to get interval_list
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Interval_List collection mounted at /keep/b34c79e916b3f4868c677e831b11c26d+94
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Total chunk length is 15618872
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Have 1 intervals
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:46:33 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 15 done, 8 running, 192 todo
2016-12-07_01:46:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:46:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:46:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:46:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:46:36 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:46:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:46:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: cpu 0.4900 user 0.1600 sys 32 cpus -- interval 10.0001 seconds 0.4900 user 0.1600 sys
2016-12-07_01:46:37 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr crunchstat: net:eth0 21877 tx 238297 rx
2016-12-07_01:46:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Have 5400 potential reusable task outputs
2016-12-07_01:46:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:46:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:46:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Have 5400 potentially reusable tasks
2016-12-07_01:46:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Creating task to process interval: [11:41537879-57156750]
2016-12-07_01:46:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr No reusable JobTask matched key parameters [u'120_of_200', u'4253ced78675bc039ec63c878e7f8d99+14700', '11:41537879-57156750', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:46:38 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 stderr Ending task 1 successfully
2016-12-07_01:46:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 killing orphaned srun process 37745 (task not in slurm queue, no stderr received in last 92s)
2016-12-07_01:46:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 sending 2x signal 2 to pid 37745
2016-12-07_01:46:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:46:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:46:39 z8ta6-8i9sb-r52unvxj6avgrio 32237  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:46:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 child 37745 on humgen-05-15.1 exit 1 success=
2016-12-07_01:46:39 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:46:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 failure (#1, temporary) after 106 seconds
2016-12-07_01:46:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 13 task output (0 bytes): 
2016-12-07_01:46:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 child 40424 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:46:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 success in 32 seconds
2016-12-07_01:46:41 z8ta6-8i9sb-r52unvxj6avgrio 32237 23 task output (0 bytes): 
2016-12-07_01:46:41 z8ta6-8i9sb-r52unvxj6avgrio 32237  2 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:46:47 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 16 done, 6 running, 194 todo
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237  Every node has failed -- giving up
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237  wait for last 6 children to finish
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr srun: Job step created
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2uxsl19t6f1g869-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2uxsl19t6f1g869-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=19 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-2uxsl19t6f1g869 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/464388cf072e0d3ea3f355afea38845a9acbe7dfd885c3273383a647a0bba46e/cgroup.procs
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/464388cf072e0d3ea3f355afea38845a9acbe7dfd885c3273383a647a0bba46e/memory.stat
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7929856 rss
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/464388cf072e0d3ea3f355afea38845a9acbe7dfd885c3273383a647a0bba46e/cpuacct.stat
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/464388cf072e0d3ea3f355afea38845a9acbe7dfd885c3273383a647a0bba46e/cpuset.cpus
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/464388cf072e0d3ea3f355afea38845a9acbe7dfd885c3273383a647a0bba46e/blkio.io_service_bytes
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Mounting task input collection to get interval_list
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Interval_List collection mounted at /keep/76a230c91cc5d0053b75306d58906b2a+94
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Total chunk length is 15498872
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Have 1 intervals
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Have 5400 potential reusable task outputs
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Have 5400 potentially reusable tasks
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Creating task to process interval: [10:130336010-135534747 11:1-10300134]
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr No reusable JobTask matched key parameters [u'117_of_200', u'a986cd76d508917b0b96bd60c6107961+14704', '10:130336010-135534747 11:1-10300134', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:46:48 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 stderr Ending task 1 successfully
2016-12-07_01:46:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 child 39557 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:46:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 success in 69 seconds
2016-12-07_01:46:49 z8ta6-8i9sb-r52unvxj6avgrio 32237 19 task output (0 bytes): 
2016-12-07_01:46:49 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-w2chjt72sypegzz-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w2chjt72sypegzz-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=25 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-w2chjt72sypegzz --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/0fe605e8e9045d3f62975d4b80e5af3076f70b191486c94fc37df26772431669/cgroup.procs
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/0fe605e8e9045d3f62975d4b80e5af3076f70b191486c94fc37df26772431669/memory.stat
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: mem 98304 cache 0 swap 0 pgmajfault 8450048 rss
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/0fe605e8e9045d3f62975d4b80e5af3076f70b191486c94fc37df26772431669/cpuacct.stat
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/0fe605e8e9045d3f62975d4b80e5af3076f70b191486c94fc37df26772431669/cpuset.cpus
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/0fe605e8e9045d3f62975d4b80e5af3076f70b191486c94fc37df26772431669/blkio.io_service_bytes
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Mounting task input collection to get interval_list
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Interval_List collection mounted at /keep/bbfdffd38f1432cef552ced110ab3a90+94
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Total chunk length is 15618872
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Have 1 intervals
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 killing orphaned srun process 38743 (task not in slurm queue, no stderr received in last 87s)
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 sending 2x signal 2 to pid 38743
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 notice: task is not in slurm queue but srun process 39552 has not exited
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 17 done, 5 running, 195 todo
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 child 38743 on humgen-05-15.6 exit 1 success=
2016-12-07_01:46:56 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:46:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 failure (#1, temporary) after 102 seconds
2016-12-07_01:46:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 15 task output (0 bytes): 
2016-12-07_01:46:58 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 17 done, 4 running, 196 todo
2016-12-07_01:46:58 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Have 5400 potential reusable task outputs
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Have 5400 potentially reusable tasks
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Creating task to process interval: [11:72775623-88394494]
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr No reusable JobTask matched key parameters [u'122_of_200', u'af62e8edc576e4baf02413c2134c9270+14700', '11:72775623-88394494', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr Ending task 1 successfully
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr srun: Job step created
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 child 41163 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 success in 26 seconds
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237 25 task output (0 bytes): 
2016-12-07_01:46:59 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-jzrtrwxfqp0jbgp-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jzrtrwxfqp0jbgp-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=24 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-jzrtrwxfqp0jbgp --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/e9a2562f6026061ee74065f7ef436a6fc3b9a34c4819ad652f90ff306c00bd3e/cgroup.procs
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/e9a2562f6026061ee74065f7ef436a6fc3b9a34c4819ad652f90ff306c00bd3e/memory.stat
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 3108864 rss
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/e9a2562f6026061ee74065f7ef436a6fc3b9a34c4819ad652f90ff306c00bd3e/cpuacct.stat
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/e9a2562f6026061ee74065f7ef436a6fc3b9a34c4819ad652f90ff306c00bd3e/cpuset.cpus
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: cpu 0.0000 user 0.0100 sys 32 cpus
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/e9a2562f6026061ee74065f7ef436a6fc3b9a34c4819ad652f90ff306c00bd3e/blkio.io_service_bytes
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Mounting task input collection to get interval_list
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Interval_List collection mounted at /keep/dcd0caa42cf81cfae2c2a7a7d6a5503a+94
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Total chunk length is 15618872
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Have 1 intervals
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:47:06 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:47:07 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 18 done, 3 running, 197 todo
2016-12-07_01:47:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:47:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:47:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:47:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:47:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:47:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24092672 rss
2016-12-07_01:47:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: cpu 0.4800 user 0.1900 sys 32 cpus -- interval 10.0002 seconds 0.4800 user 0.1800 sys
2016-12-07_01:47:10 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr crunchstat: net:eth0 21919 tx 238061 rx
2016-12-07_01:47:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 notice: task is not in slurm queue but srun process 40085 has not exited
2016-12-07_01:47:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Have 5400 potential reusable task outputs
2016-12-07_01:47:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:47:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:47:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Have 5400 potentially reusable tasks
2016-12-07_01:47:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Creating task to process interval: [11:57156751-72775622]
2016-12-07_01:47:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr No reusable JobTask matched key parameters [u'121_of_200', u'7770dae0a3517e6dca33c49e54e80074+14704', '11:57156751-72775622', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:47:11 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 stderr Ending task 1 successfully
2016-12-07_01:47:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr srun: Job step created
2016-12-07_01:47:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 child 40806 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:47:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 success in 51 seconds
2016-12-07_01:47:12 z8ta6-8i9sb-r52unvxj6avgrio 32237 24 task output (0 bytes): 
2016-12-07_01:47:12 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-g3qgtrm3ymgzwry-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-g3qgtrm3ymgzwry-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=18 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-g3qgtrm3ymgzwry --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/87f19283681a04439576592825f1a501b3f32a2f59e9202ac3fa3ea420743645/cgroup.procs
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/87f19283681a04439576592825f1a501b3f32a2f59e9202ac3fa3ea420743645/memory.stat
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7929856 rss
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/87f19283681a04439576592825f1a501b3f32a2f59e9202ac3fa3ea420743645/cpuacct.stat
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/87f19283681a04439576592825f1a501b3f32a2f59e9202ac3fa3ea420743645/cpuset.cpus
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/87f19283681a04439576592825f1a501b3f32a2f59e9202ac3fa3ea420743645/blkio.io_service_bytes
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Mounting task input collection to get interval_list
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Interval_List collection mounted at /keep/359a96b521b56e8d8e36b57450fa80ac+94
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Total chunk length is 15618872
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Have 1 intervals
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:47:18 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 19 done, 2 running, 198 todo
2016-12-07_01:47:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Have 5400 potential reusable task outputs
2016-12-07_01:47:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:47:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:47:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Have 5400 potentially reusable tasks
2016-12-07_01:47:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Creating task to process interval: [10:114717138-130336009]
2016-12-07_01:47:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr No reusable JobTask matched key parameters [u'116_of_200', u'6365525c0ee094fee362794dbbab6a30+14700', '10:114717138-130336009', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:47:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 stderr Ending task 1 successfully
2016-12-07_01:47:21 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr srun: Job step created
2016-12-07_01:47:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 child 39552 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:47:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 success in 101 seconds
2016-12-07_01:47:22 z8ta6-8i9sb-r52unvxj6avgrio 32237 18 task output (0 bytes): 
2016-12-07_01:47:22 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-g6whygrqqg4quk3-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-g6whygrqqg4quk3-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=22 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-g6whygrqqg4quk3 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8a8378d7ebf8af79372a73d14f3f74501673f5004cc60177e812d07530703d65/cgroup.procs
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8a8378d7ebf8af79372a73d14f3f74501673f5004cc60177e812d07530703d65/memory.stat
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4878336 rss
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8a8378d7ebf8af79372a73d14f3f74501673f5004cc60177e812d07530703d65/cpuacct.stat
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8a8378d7ebf8af79372a73d14f3f74501673f5004cc60177e812d07530703d65/cpuset.cpus
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8a8378d7ebf8af79372a73d14f3f74501673f5004cc60177e812d07530703d65/blkio.io_service_bytes
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Mounting task input collection to get interval_list
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Interval_List collection mounted at /keep/a19791935337560ca00961b9d0c50277+92
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Total chunk length is 15618872
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Have 1 intervals
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:47:28 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:47:29 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 20 done, 1 running, 199 todo
2016-12-07_01:47:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:47:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:47:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:47:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:47:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:47:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24092672 rss
2016-12-07_01:47:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: cpu 0.5200 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4900 user 0.1400 sys
2016-12-07_01:47:32 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr crunchstat: net:eth0 21877 tx 238215 rx
2016-12-07_01:47:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Have 5400 potential reusable task outputs
2016-12-07_01:47:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:47:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:47:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Have 5400 potentially reusable tasks
2016-12-07_01:47:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Creating task to process interval: [1:156188721-171807592]
2016-12-07_01:47:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr No reusable JobTask matched key parameters [u'11_of_200', u'0407d47186859c17195bd2efb49797bb+14616', '1:156188721-171807592', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:47:33 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 stderr Ending task 1 successfully
2016-12-07_01:47:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 child 40085 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:47:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 success in 96 seconds
2016-12-07_01:47:34 z8ta6-8i9sb-r52unvxj6avgrio 32237 22 task output (0 bytes): 
2016-12-07_01:47:34 z8ta6-8i9sb-r52unvxj6avgrio 32237  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:47:40 z8ta6-8i9sb-r52unvxj6avgrio 32237  Got 1 new tasks from API server.
2016-12-07_01:47:41 z8ta6-8i9sb-r52unvxj6avgrio 32237  status: 21 done, 0 running, 200 todo
2016-12-07_01:47:41 z8ta6-8i9sb-r52unvxj6avgrio 32237  collate
2016-12-07_01:47:41 z8ta6-8i9sb-r52unvxj6avgrio 32237  collated output manifest text to send to API server is 0 bytes with access tokens
2016-12-07_01:47:41 z8ta6-8i9sb-r52unvxj6avgrio 32237  job output d41d8cd98f00b204e9800998ecf8427e+0