Arvados

Suggest checking refresh_trigger (and the other things) every ~two seconds, even during times when readfrompipes() and reapchildren() are keeping us 100% busy.

-    if (!$gotsome) {
+    if (!$gotsome || $latest_refresh + 2 < scalar time) {

(I'm not confident this will fix the issue at hand, but it's a good idea anyway.)

FWIW I canceled a few jobs recently and it worked as expected. I'd want to test with a job that has lots of parallel tasks before declaring victory though.

I should note that I just updated the API server Gemfile pin on arvados-cli, which previously was 3 months out of date. So the specific fix made for this ticket may or may not be in production yet.

Peter Amstutz wrote:

I should note that I just updated the API server Gemfile pin on arvados-cli, which previously was 3 months out of date. So the specific fix made for this ticket may or may not be in production yet.

Our clusters run crunch-dispatch with CRUNCH_JOB_BIN set to point to a copy of crunch-job from the arvados-src package. So it uses the version deployed through that package, regardless of what the API server's Gemfile says.

Actually, the number of tasks (400) was exactly equal to the number of cores on the allocated nodes (10 nodes with 40 cores each). I'm not running any compute on the head node, so all 40 cores there are available for the API server, crunch-dispatch, etc.

I manually changed the period for crunchstat from 10s to 300s and ran it again and this time it was able to cancel the job (although it took ~4m to completely finish being cancelled and free up the nodes).

Just what happened today as an example:

API server got rebooted (so job dispacher) but slurm controller is running in manage.wx7k5 then the state ended up as:

compute0.wx7k5:/home/nico# sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite     34 drain* compute[2-35]
compute*     up   infinite    220  down* compute[36-255]
compute*     up   infinite      2  alloc compute[0-1]
crypto       up   infinite     34 drain* compute[2-35]
crypto       up   infinite    220  down* compute[36-255]
crypto       up   infinite      2  alloc compute[0-1]
compute0.wx7k5:/home/nico# squeue_long 
  JOBID PARTITION NAME     USER ST       TIME  NODES NODELIST(REASON)
    868   compute wx7k5-8i9sb-258pea56x155uqa   crunch  R   21:14:04      2 compute[0-1]

but no docker container running:

compute0.wx7k5:/home/nico# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

the solution was

wx7k5:~# scancel 868

should we have a way to do an scancel from workbench?

I've merged tomclegg's 7263-better-busy-behavior with the substantial changes I've made to our crunch-job for performance improvements to address this issue.

The fixes in my branch are:

readfrompipes:
- Incorporates the "fix II" from this issue
- Also changes the loop to use `can_read` from IO::Select so that that we only loop over descriptors that are actually ready for reading rather than wasting time calling sysread on a large number of descriptors that are not ready for reading - this is a big performance improvement with large numbers of tasks in a situation where a small number of tasks are producing most of the stderr and also in the non-busy situation.

reapchildren:
- changes behavior of reapchildren from reaping a single child to actually reaping children (i.e. continues to reap children until there are no children ready to be reaped).
- fixes bug where reapchildren did not check WIFEXITED, which could cause reapchildren to mistakenly mark tasks as exited that had only stopped (on some platforms; not actually observed, but this is best practice)

check_refresh_wanted:
- Fixes the broken handling of "%proc" values as "$jobstep"when actually one needs to access "$_->{jobstep}"to get "$jobstep" from "%proc" values.

https://github.com/wtsi-hgi/arvados/compare/hgi/7263-better-busy-behavior...wtsi-hgi:hgi/7263-even-better-busy-behavior

7263-better-busy-behavior and f9d8685 looks good to me.

7263-better-busy-behavior LGTM. I've merged it into 8099-babysit-all-srun (#8099) which touched some of the same bits (including renaming $jobstep to $jobstepidx throughout to avoid some of the confusion that probably made us write these bugs in the first place) so I could try it all together on a test cluster.

Now just one new commit 0f7709af on wtsi-hgi:hgi/7263-even-better-busy-behavior to review/merge, see #8575. The rest of that branch was merged into #8099, and from there into master.