Arvados

Currently, a container that has state==Cancelled when arvados-dispatch-cloud starts up will never be added to the container queue, even if its UUID appears on an instance's probe result. Also, a container that has been deleted from the database will never have an entry added/updated in the dispatcher's container queue.

The scheduler's sync() func is responsible for killing unneeded crunch-run processes, but it only looks at the container queue, so these crunch-run processes are allowed to run forever.

Proposed solution:

In (*scheduler.Scheduler)sync(), kill anything returned by sch.pool.Running() that isn't returned by sch.queue.Entries(). This should be safe from "kill crunch-run before seeing its UUID in the queue" races:

• at least one "get entire queue from controller/database" has succeeded before the first call to sync()
• UUIDs are added to Running() only during (*Scheduler)runQueue(), which does not run concurrently with (*Scheduler)sync().

In (*container.Queue)poll(), if a container's UUID is in the local queue but is not returned by the API calls that request that specific UUID, delete it from the local queue. The "get missing containers" loop will need to be more careful to avoid accidentally deleting containers when the API server chooses to return less than a full page of results.