Arvados

This state occurs if the node is allocated before the drain requests goes through. This can happen if Node Manager simply loses the race with crunch-dispatch, or if something interferes with the drain request like #6321.

Once the node is allocated, it will no longer be eligible for shutdown, and Node Manager will try to cancel the pending node shutdown. The first step of that is resuming the node in SLURM—but that can't succeed if the node isn't already drained. So that request fails, and then Node Manager's state is bad.

review @ 2a94b125b93a3aba204f55c37ecdc2876d81d642

I looked at the code and I ran the tests (all passed). This is new code for me, It's not evident for my current knowledge to have a major problem here.

LGTM