Arvados

• If the cluster is set up with shared home directories (which is common in slurm setups) then the messages might appear in $HOME/slurm-{jobid}.out
• If we know which compute node tried to run the job, we might be able to run something like "srun --immediate --share --nodelist=computeX cat slurm-{jobid}.out"
  • If we don't know which node to look on, we might be able to look on all nodes using "srun --immediate --share -N{number_of_nodes} cat slurm-{jobid}.out".
  • Either way, success depends on cluster configuration. "The default shared behavior depends on system configuration and the partition's Shared option takes precedence over the job's option."
• In the script we pass to sbatch, check for curl and wget -- if one exists, use it to do a "create log" API call using the contents of slurm-*.out
• Use a bash trick to do an HTTP POST? http://unix.stackexchange.com/questions/83926/how-to-download-a-file-using-just-bash-and-nothing-else-no-curl-wget-perl-et

We can also try running "srun" with the same flags, since it is more likely to return useful errors from the subprocess.