Arvados

It looks like the last page received had 3 items, and the repeated timestamp is very new, so presumably the reason we got 3 items was that there are no more recent ones to get.

I think this means we need an exception to that starvation check: it's perfectly fine to receive a page of identical timestamps if we don't need to get the next page (i.e., if there is no next page).

Tom Clegg wrote:

it's perfectly fine to receive a page of identical timestamps if we don't need to get the next page (i.e., if there is no next page).

We already have this check:

        if last.ModifiedAt == nil || *last.ModifiedAt == filterTime {
            if page.ItemsAvailable > len(page.Items) {
                // TODO: use "mtime=X && UUID>Y" 
                // filters to get all collections with
                // this timestamp, then use "mtime>X" 
                // to get the next timestamp.
                return fmt.Errorf("BUG: Received an entire page with the same modified_at timestamp (%v), cannot make progress", filterTime)
            }
            break
        }

I wonder if this is a race on the API server, when a collection is added or modified between the "get items" query and the "count items for items_available" query, so they don't reflect the same database state.

See https://www.postgresql.org/docs/9.1/static/transaction-iso.html → 13.2.2. Repeatable Read Isolation Level

• "Applications using this level must be prepared to retry transactions due to serialization failures."
• "Note that only updating transactions might need to be retried; read-only transactions will never have serialization conflicts."

We should also implement the "mtime=X and UUID>Y" fallback described in the comment, like we did in datamanager. This is useful even with the transaction behavior fixed: it will still be possible to get a full page of results with identical timestamps.