Arvados

If I am running Arvados on local hardware, it's possible to consume storage at an alarming rate. For example, I am working with LLMs that are 100+ gigabytes and end up with multiple copies resulting from intermediate processing steps. In three weeks I've already consumed 70% of a 3 TiB partition.

I have two problems.

• I want to preserve provenance, but if I delete intermediate collections, I'll lose that. My idea is to be able to discard the underlying data while keeping the collection record around (#22458)

• The normal process of putting stuff in the trash means no storage is reclaimed for 2+ weeks. This sucks and as a user I could see myself being very unhappy if I have a crises where I have run out of space, need to do work, and even after going and trashing things I'm willing to part with, no actual storage space has been reclaimed.

We should have "empty trash NOW" mode for the API server/keep-balance/keepstore, which ignores the normal grace periods and aggressively deletes everything it can.

No containers should be allowed to run and no data can be written to keep while the "trash now" process is happening.

Write a design plan for this feature.