Feature #18180
closed
Ability to control use of spot instances on a per-workflow and step level
Added by Peter Amstutz about 3 years ago.
Updated almost 3 years ago.
Estimated time:
(Total: 0.00 h)
Release relationship:
Auto
Description
- container_request scheduling_parameter['preemptible'] doesn't have to match the setting of the cluster
- ability to choose spot instances when submitting workflow with arvados-cwl-runner
- --enable-preemptible --disable-preemptible
- gets passed through to the arvados-cwl-runner leader instance
- --disable-preemptible opts out for whole workflow (overrides)
- --enable-preemptible opts in for the whole workflow
- Individual steps can opt out if preemptible is enabled
Related issues
1 (1 open — 0 closed)
- Description updated (diff)
- Category set to Crunch
- Description updated (diff)
- Target version set to 2022-03-16 sprint
- Assigned To set to Peter Amstutz
- Subject changed from Ability to control use of spot instances on a per-workflow basis to Ability to control use of spot instances on a per-workflow and step level
- Description updated (diff)
- Target version changed from 2022-03-16 sprint to 2022-03-30 Sprint
Code looks good to me. Here're my doc-related comments:
- Files
arv-cwl-schema*.yml: The fields.class.doc keys seems to need fixing. (Copy&pasted content from other extension?)
- Documentation at
/admin/spot-instances.html might need some update: "When UsePreemptibleInstances is enabled, child containers (workflow steps) will automatically be made preemptible..."
- I think the doc pages
/user/cwl/cwl-runner.html & /user/cwl/cwl-run-options.html could also need some updates regarding this new feature.
Lucas Di Pentima wrote:
Code looks good to me. Here're my doc-related comments:
- Files
arv-cwl-schema*.yml: The fields.class.doc keys seems to need fixing. (Copy&pasted content from other extension?)
Fixed.
- Documentation at
/admin/spot-instances.html might need some update: "When UsePreemptibleInstances is enabled, child containers (workflow steps) will automatically be made preemptible..."
Added more text about it.
- I think the doc pages
/user/cwl/cwl-runner.html & /user/cwl/cwl-run-options.html could also need some updates regarding this new feature.
Updated cwl-run-options and cwl-style.
While I was at it, I also improved the formatting of cwl-style a bit.
18180-cwl-spot @ f21f62d680e200d113d6fd700db743e702e96602
- Status changed from New to Resolved
I noticed that the "Scheduling parameters" section at doc/api/methods/containers.html says
| key |
type |
description |
notes |
| preemptible |
boolean |
If true, the dispatcher will ask for a preemptible cloud node instance (eg: AWS Spot Instance) to run this container. |
Optional. Default is false. |
Isn't that default more subtle now, depending on the cluster configuration?
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by