Arvados

The following sequence can break the invariant Keep relies on for safe garbage collection¹:

1. Write data blocks
2. Save collection
3. Wait for original blob signature TTLs (from step 1) to expire
4. Retrieve the collection (setting aside a copy in memory)
5. Update the collection to remove references to some (or all) blocks
6. Wait for keep-balance to do a garbage-collection sweep
7. Update the collection (or save a new one) using the blocks/signatures set aside in step 4

If the only reference to a block is the one in this collection, the block may be deleted during step 6, resulting in a collection being saved successfully in step 7 even though some of its data is missing.

Proposed solution:

When updating a collection, detect block references that are being dropped²; if there are any, save a temporary collection (trashed, readable only by admin) referencing all of the dropped blocks. These will be cleaned up when they expire, and in the meantime they will be seen by keep-balance, thus protecting them until even their new signatures (issued in step 4) expire.

¹ If any client has a signed block locator with a future timestamp, that block either [i] is newer than BlobSignatureTTL according to the timestamp stored on the backend, and therefore won't be deleted; or [ii] appears in a collection record where keep-balance can see it, and therefore won't be deleted.

² This might be done during the (existing) strip_signatures_and_update_replication_confirmed hook where we detect whether references to new blocks have been added, so we know whether to reset the replication_confirmed columns.