Arvados

I am re-using this ticket, since we had a lot of inquiries about this. This is an example of it:

We are trying to delete data from the keepstore. But after we run "arv collection delete --uuid=zzzzz-4zz18-7p9s7j1qa" it trashes it, but sets the final delete date 2 weeks in the future.

I ask because on our test server we have filled up the keepstore and can't delete data to continue testing. It's put a stop to the project.

 "delete_at":"2019-07-11T20:45:30.557593000Z",
 "trash_at":"2019-06-27T20:45:30.557593000Z",
 "is_trashed":true,

Even if we change the trash times in " /etc/arvados/keepstore/keepstore.yml" and reinstall keepstore + restart the service.

\# How often to check for (and delete) trashed blocks whose

 # TrashLifetime has expired.
 TrashCheckInterval: 1h0m0s

 # Time duration after a block is trashed during which it can be
 # recovered using an /untrash request.
 TrashLifetime: 1h0m0s

this was my answer:

Just to make sure we are all in the same page in terms of terminology, there is a good explanation here: https://doc.arvados.org/user/tutorials/tutorial-keep-collection-lifecycle.html

And the method you used is "delete" in collections, from https://doc.arvados.org/v1.4/api/methods/collections.html
----------
delete

Put a Collection in the trash. This sets the trash_at field to now and delete_at field to now + token TTL. A trashed collection is invisible to most API calls unless the include_trash parameter is true.
-----------

As you can see the "token TTL" expressed there is set by default in 2 weeks, this is the Collections->BlobSigningTTL and Collections->DefaultThrashLifetime parameters in the configuration. Here is description from in https://doc.arvados.org/v1.4/admin/config.html

-------

# Lifetime (in seconds) of blob permission signatures generated by
# the API server. This determines how long a client can take (after
# retrieving a collection record) to retrieve the collection data
# from Keep. If the client needs more time than that (assuming the
# collection still has the same content and the relevant user/token
# still has permission) the client can retrieve the collection again
# to get fresh signatures.
#
# This must be exactly equal to the -blob-signature-ttl flag used by
# keepstore servers.  Otherwise, reading data blocks and saving
# collections will fail with HTTP 403 permission errors.
#
# Modifying blob_signature_ttl invalidates existing signatures; see
# blob_signing_key note above.
#
# The default is 2 weeks.
BlobSigningTTL: 336h

# Default lifetime for ephemeral collections: 2 weeks. This must not
# be less than blob_signature_ttl.
DefaultTrashLifetime: 336h

------

This assumes that you have the central configuration in /etc/arvados/config.yml, keep-balance.service up and running.

As you can see we have 3 different places with pieces of the information. And usually our test server we have filled up the keepstore is the reason they need a quick "delete all this" process without having to wait 2 weeks.

One of the main things that's missing from https://doc.arvados.org/user/tutorials/tutorial-keep-collection-lifecycle.html (which is really conceptual documentation, not a tutorial)
is "when do I get my disk space back?" and the associated Keep store pieces of the data lifecycle.

11016-doc-signing-ttl @ f902c301c8c40707bb25f8dae01537f8a50be286 -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1571/

• add "(ie Time-To-Live)" after "BlobSigningTTL determines the minimum lifetime" to help explain TTL. I'm not sure it's a familiar term outside of the networking space.
• it seems like there are pieces of the lifecycle for blocks and signed block references missing from https://doc.arvados.org/user/tutorials/tutorial-keep-collection-lifecycle.html unless we have them covered somewhere else (ie the two cases mentioned in the config file description)