Arvados

as a deploy/upgrade procedure of a cluster, So the situation is the following:

• I send a TERM signal to crunch dispatch jobs to set a DRAIN mode
• a user sends a pipeline instance to run
• nodemanager creates nodes based on the queue this nodes stay up forever

the collateral problem to this is that those nodes will up and running with an OLD version of the compute image. After the upgrade of all components we need to kill all the nodes so nodemanager can spin new ones.

There are several solutions here:

1. have a flag in the API server (or Consul in the future) called "draining" that nodemanager and crunch will query, changing their behavior
2. have some other type of signaling (like a TERM signal) to nodemanager to implement the draining mode
3. both.