Arvados

Seen in pirca-xvhdp-eg0f3jykdrzdsf7:

2025-05-22T18:15:08.060Z Waiting in queue at position 1. Cluster is at capacity and cannot start any new instances right now.
2025-05-22T18:15:29.078Z Waiting in queue at position 1. Cluster is at capacity and cannot start any new instances right now.
2025-05-22T18:15:49.220Z Waiting in queue at position 1. Cluster is at capacity and cannot start any new instances right now.
2025-05-22T18:16:09.515Z Waiting in queue at position 1. Cluster is at capacity and cannot start any new instances right now.
2025-05-22T18:16:29.661Z Waiting in queue at position 1. Cluster is at capacity and cannot start any new instances right now.
2025-05-22T18:16:49.938Z Container is allocated to an instance and preparing to run.
2025-05-22T18:16:39.697143267Z using local keepstore process (pid 1447) at http://10.254.254.180:45001, writing logs to keepstore.txt in log collection
2025-05-22T18:16:42.775935318Z gateway server listening at 10.254.254.180:35075
2025-05-22T18:16:42.776099213Z crunch-run 3.1.2~rc2 (go1.23.6) started

It seems odd that we went directly from "cannot start an instance" to "allocated to an instance" in 20 seconds without any "booting an instance" messages in the middle. I can imagine circumstances where that happens but I only had two active workflows on pirca at the time so I can't imagine any reason why those would've come up at the time.