Arvados

If crunch-job tries to run a Docker job on a compute node that doesn't have crunchrunner installed, it passes an empty string as an argument to docker run. That causes the job to fail like this:

2016-04-04_15:28:16 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr Running [docker.io run --name=qr2hi-ot0gb-shrdsi503141h6h-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/tmp/crunch-job/qr2hi-ot0gb-shrdsi503141h6h-0.cid --sig-proxy --memory=3610756k --memory-swap=3610756k --volume=/tmp/crunch-job/src:/tmp/crunch-job/src:ro --volume=/tmp/crunch-job/opt:/tmp/crunch-job/opt:ro --volume=/tmp/crunch-job/task/compute10.1.keep/by_pdh:/keep:ro --volume=/tmp/crunch-job/task/compute10.1.keep/tmp:/keep_tmp --volume=/tmp  --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_BIN=/usr/local/arvados/src/services/crunch/crunch-job --env=TASK_SEQUENCE=0 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_SAMPLE-NAMES=Unstimulated,LPS_6h --env=CRUNCH_SRC_COMMIT=766af589846b703e39bc1bd4fd7051fffb6069c8 --env=JOB_PARAMETER_RSEM-CONTROL-FDR-ARGS= --env=TASK_QSEQUENCE=0 --env=CRUNCH_INSTALL=/tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4049a8b0f3d36eb5de13b929902148be --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=1u5pnwcp0ns12zzooyvvhm9wvhw80f9qfczyshofaj2z47vmew --env=CRUNCH_WORK=/tmp/crunch-job/work --env=CRUNCH_TMP=/tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/compute10.1 --env=JOB_UUID=qr2hi-8i9sb-qjzfzlu629zm7ux --env=JOB_PARAMETER_FDR=0.05 --env=CRUNCH_JOB_UUID=qr2hi-8i9sb-qjzfzlu629zm7ux --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=compute10 --env=JOB_PARAMETER_INPUT=a0d8315f3545588599deba590e0bd1aa+958 --env=JOB_SCRIPT=run_ebseq.py --env=CRUNCH_NODE_SLOTS=1 --env=TASK_WORK=/tmp/crunch-job-task-work/compute10.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=qr2hi.arvadosapi.com --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=qr2hi-ot0gb-shrdsi503141h6h --env=CRUNCH_SRC=/tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/compute10.1 a1b9ed39b2a59d6e76164d66467c0912afcee3d40d622b103cb7e4d8cca2d5c3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/compute10.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/tmp\/crunch\-job\/src\/crunch_scripts\/run_ebseq\.py ; else   exec \/tmp\/crunch\-job\/src\/crunch_scripts\/run_ebseq\.py ; fi]
2016-04-04_15:28:16 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr Usage:\011docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
2016-04-04_15:28:16 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr Run a command in a new container
2016-04-04_15:28:16 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -a, --attach=[]                 Attach to STDIN, STDOUT or STDERR
2016-04-04_15:28:16 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --add-host=[]                   Add a custom host-to-IP mapping (host:ip)
2016-04-04_15:28:16 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --blkio-weight=0                Block IO (relative weight), between 10 and 1000
2016-04-04_15:28:16 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -c, --cpu-shares=0              CPU shares (relative weight)
2016-04-04_15:28:16 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --cap-add=[]                    Add Linux capabilities
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --cap-drop=[]                   Drop Linux capabilities
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --cgroup-parent=                Optional parent cgroup for the container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --cidfile=                      Write the container ID to the file
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --cpu-period=0                  Limit CPU CFS (Completely Fair Scheduler) period
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --cpu-quota=0                   Limit CPU CFS (Completely Fair Scheduler) quota
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --cpuset-cpus=                  CPUs in which to allow execution (0-3, 0,1)
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --cpuset-mems=                  MEMs in which to allow execution (0-3, 0,1)
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -d, --detach=false              Run container in background and print container ID
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --device=[]                     Add a host device to the container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --disable-content-trust=true    Skip image verification
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --dns=[]                        Set custom DNS servers
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --dns-search=[]                 Set custom DNS search domains
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -e, --env=[]                    Set environment variables
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --entrypoint=                   Overwrite the default ENTRYPOINT of the image
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --env-file=[]                   Read in a file of environment variables
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --expose=[]                     Expose a port or a range of ports
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --group-add=[]                  Add additional groups to join
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -h, --hostname=                 Container host name
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --help=false                    Print usage
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -i, --interactive=false         Keep STDIN open even if not attached
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --ipc=                          IPC namespace to use
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -l, --label=[]                  Set meta data on a container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --label-file=[]                 Read in a line delimited file of labels
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --link=[]                       Add link to another container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --log-driver=                   Logging driver for container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --log-opt=[]                    Log driver options
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --lxc-conf=[]                   Add custom lxc options
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -m, --memory=                   Memory limit
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --mac-address=                  Container MAC address (e.g. 92:d0:c6:0a:29:33)
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --memory-swap=                  Total memory (memory + swap), '-1' to disable swap
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --memory-swappiness=-1          Tuning container memory swappiness (0 to 100)
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --name=                         Assign a name to the container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --net=default                   Set the Network mode for the container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --oom-kill-disable=false        Disable OOM Killer
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -P, --publish-all=false         Publish all exposed ports to random ports
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -p, --publish=[]                Publish a container's port(s) to the host
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --pid=                          PID namespace to use
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --privileged=false              Give extended privileges to this container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --read-only=false               Mount the container's root filesystem as read only
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --restart=no                    Restart policy to apply when a container exits
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --rm=false                      Automatically remove the container when it exits
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --security-opt=[]               Security Options
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --sig-proxy=true                Proxy received signals to the process
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -t, --tty=false                 Allocate a pseudo-TTY
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -u, --user=                     Username or UID (format: <name|uid>[:<group|gid>])
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --ulimit=[]                     Ulimit options
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --uts=                          UTS namespace to use
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -v, --volume=[]                 Bind mount a volume
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --volume-driver=                Optional volume driver for the container
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   --volumes-from=[]               Mount volumes from the specified container(s)
2016-04-04_15:28:17 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr   -w, --workdir=                  Working directory inside the container
2016-04-04_15:28:22 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 stderr crunchstat: error reading cid file: /tmp/crunch-job/qr2hi-ot0gb-shrdsi503141h6h-0.cid
2016-04-04_15:28:22 qr2hi-8i9sb-qjzfzlu629zm7ux 10164 0 child 10661 on compute10.1 exit 0 success=