Story #3762
closed[Keep] Process entries on the current "delete" list.
100%
Description
Keep should process the file asynchronously and delete each block in it.
This is written out in more detail at the bottom of:
https://arvados.org/projects/orvos-private/wiki/Keep_Design_Doc#Keep-Server
- For now, begin processing the trash list right away. We can defer to a future sprint how Keep should decide when to start recycling blocks.
- If a block's actual age is less than (some delta) it should not be deleted.
Need to define how this threshold is chosen. (Ideally this is defaultTrashLifetime from the discovery doc, but a command line flag would be OK.)"Some delta" is the value of the existing-permission-ttl
flag. If a new list is received while the old one is being processed, abandon the existing one, replace it with the new one, and start over.(WorkQueue already takes care of this.)
Updated by Tom Clegg over 10 years ago
- Description updated (diff)
- Category set to Keep
Updated by Tom Clegg almost 10 years ago
- Target version changed from Arvados Future Sprints to 2015-03-11 sprint
Updated by Tom Clegg almost 10 years ago
- Target version changed from 2015-03-11 sprint to Arvados Future Sprints
Updated by Tom Clegg almost 10 years ago
- Target version changed from Arvados Future Sprints to 2015-04-01 sprint
Updated by Radhika Chippada almost 10 years ago
- Assigned To set to Radhika Chippada
Updated by Radhika Chippada almost 10 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz almost 10 years ago
Reviewing 3762-delete-list-worker fed6d3b
The cached discovery document should part of the ArvadosClient struct, not the DISCOVERY
global variable. You should probably fetch the discovery document in MakeArvadosClient().
If you want to return "any type" in Go, you use interface{}
. Then Discovery()
does not need to create valueMap just to return as single item.
I just realized that having the trashWorker fetch "defaultTrashLifetime" from the discovery document means that we are re-introducing a dependency on the API server that we had just worked to eliminate with the pullWorker. We should consult tom/ward on their preference.
I noticed this in both pullq and trashq:
if pullq == nil { pullq = NewWorkQueue() }
trashq
and pullq
should never be nil, and if they are nil, allocating a new work queue won't do anything because there won't be a worker goroutine associated with it. These tests should be taken out or converted into a assertions that raise an error.
Updated by Radhika Chippada almost 10 years ago
- Removed all the new code for API discovery and instead using permission_ttl. Per our discussion with Tom, defaultTrashLifetime is the same as permission_ttl, which is already available in keepstore and this should be used.
- Removed "if pullq == nil ... " from handlers.go and updated the handler_test instead to create the instances of pullq and trashq.
Updated by Radhika Chippada almost 10 years ago
- Status changed from In Progress to Resolved
- % Done changed from 50 to 100
Applied in changeset arvados|commit:0d1971f04c8f0674ff8d94bae5dbb7f6b94e7f5f.