Arvados

"wardv" is lexicographically later than "ward2", but it was added before "ward2", so it has a lower id.

The find_usable_username_from method tries to do this:

select username from users where username like 'ward_' order by username;
=> ward2, ward3, ...

However, find_each() is incompatible with order and limit. Instead of returning an error, it emits a log message and ignores the requested sort order. So we get this:

select username from users where username like 'ward_' order by id;
=> wardv, ward2, ...

And in the logs:

Scoped order and limit are ignored, it's forced to be batch order and batch size

("batch order" means numeric id -- it's not something we can override like we can with batch size.)

We see "wardv" before seeing "ward2", we assume "ward2" doesn't exist (because we asked for sorted results!) and try to use it.

It seems the easy fix is to use each instead of find_each. The default find_each batch size is 1000 so this won't need more memory than find_each until we exceed 2000 wards.