Arvados

Before deleting a block, in order to detect GC races, keepstore's trash worker compares the timestamp of the stored block to the timestamp provided by keep-balance in the trash list, which in turn comes from keepstore's "index" worker.

The stored block's timestamp comes from the Last-Modified header from an S3 "head object" response, which is RFC1123-formatted and therefore has no better than 1-second precision.

The index worker's timestamp comes from the LastModified XML tag in the S3 "list objects" response, which is RFC3339-formatted and potentially has a non-zero nanosecond part.

AWS and Google implementations of S3 always return zero nanoseconds in the "list objects" response, but Ceph/nautilus returns timestamps with milliseconds, which as a result will never match the RFC1123-formatted timestamp used by the trash worker.

To resolve this, the S3 index handler should truncate the nanosecond part when parsing timestamps.