Arvados

Las week we had some infrastructure issues causing our API to fail. This led to several crunch jobs not being properly finalised and remaining in "Running" for a week even if they only had less than an hour of runtime. This delayed the resolution and ultimately delivery of result data by several days.

Apparently crunch failed to update the API and then tried to transition the container request from running to queued, which seems to be an invalid transition. As a result the step remained in an seemingly running state.

It would be good if a situation like this would not lead to such a situation and Arvados would handle this more gracefully.

Workflow: https://wb2.arkau.roche.com/processes/arkau-xvhdp-u7racu94jfsqxn5