Arvados

Problem¶

There are circumstances where crunch-dispatch-* tries to run a container, but something fails before the container gets to Running state, so the container goes back to Queued state. See #9679.

If the same problem keeps happening, the container just flaps between Queued and Locked.

After a certain amount of time, or number of retries, we should really just give up and cancel the container.

Proposed solutions¶

Pick one of:

• (crunch-dispatch-*) If a single container gets dispatched more than N times (over a period of at least M seconds) by a single crunch-dispatch-* process, but still won't run, give up and change state to Cancelled.
• (API server) If a container has been Locked and returned to Queued state, and is more than M seconds old, cancel it.

Also (optional), to mitigate starvation risk in a multiple-dispatch setup:

• Introduce a delay between "return container X to queue" and "re-attempt X".