Feature #19847
closed
Choose disk cache size based on RAM request
100%
Description
Instead of applying a flat 8 GB disk cache size it should be proportional to the task size.
After giving it some thought, let's try
2 GiB <= the size of the RAM request <= 32 GiB
Updated by Peter Amstutz about 2 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz about 2 years ago
- Subject changed from Choose disk cache size based on ram/cpu count to Choose disk cache size based on RAM request
- Description updated (diff)
Updated by Peter Amstutz about 2 years ago
19847-cwl-disk-cache-size @ 0531cc9f35e78a7be1eec7eb96fb6cc668ebcefa
- Default disk cache heuristic computed by arv-mount is now
2 GiB <= the size of the RAM request <= 32 GiB - Added arvados-cwl-runner CWL extension to explicitly select between ram and disk cache
- Fixed crunch-run to pass
--ram-cacheto explicitly select RAM cache when specified by runtime_constraints
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-12-07 Sprint to 2022-12-21 Sprint
Updated by Tom Clegg about 2 years ago
This seems to mean the DefaultKeepCacheDisk config is essentially a boolean (all non-zero values just mean "enable disk cache, choose size automatically") when it comes to containers submitted by arvados-cwl-runner -- is that right? Would this be worth mentioning in the config comment?
Updated by Peter Amstutz about 2 years ago
Tom Clegg wrote in #note-6:
This seems to mean the
DefaultKeepCacheDiskconfig is essentially a boolean (all non-zero values just mean "enable disk cache, choose size automatically") when it comes to containers submitted by arvados-cwl-runner -- is that right? Would this be worth mentioning in the config comment?
If I add that to the config comment would this be a LGTM?
Updated by Tom Clegg about 2 years ago
- rename config to
MinimumKeepCacheDisk, instead of using hard-coded 2 GiB in a-c-r? - config sets the default disk cache size as % RAM size instead of # bytes?
- auto-size logic moves to railsapi so it applies to top-level a-c-r containers too?
Updated by Tom Clegg about 2 years ago
would you be able to implement the behavior you want on the rails api side and then I'll update the a-c-r branch?
I did both sides, but kept the a-c-r updates in a separate commit.
19847-cwl-disk-cache-size @ 2c8bbcd431eb3aee1b4fd650b8268979b8da556c -- developer-run-tests: #3421
retry wb1 developer-run-tests-apps-workbench-integration: #3680
Updated by Peter Amstutz about 2 years ago
Tom Clegg wrote in #note-9:
would you be able to implement the behavior you want on the rails api side and then I'll update the a-c-r branch?
I did both sides, but kept the a-c-r updates in a separate commit.
19847-cwl-disk-cache-size @ 2c8bbcd431eb3aee1b4fd650b8268979b8da556c -- developer-run-tests: #3421
retry wb1 developer-run-tests-apps-workbench-integration: #3680
Your implementation LGTM, if you're happy I'm happy.
Updated by Peter Amstutz about 2 years ago
- Status changed from In Progress to Resolved