Idea #8748
closed
[Crunch] All pipeline outputs should go to the parent project, regardless of success or failure
Description
The fact that the output of failed pipelines (the logs and output files) are uploaded into the home directory means that sharing projects containing failed pipelines for debugging purposes isn't very effective.
Could we update the saving functionality to upload the logs and output of a failed pipeline into the project that owns the pipeline?
Updated by Brett Smith almost 10 years ago
- Tracker changed from Bug to Idea
- Subject changed from [Workbench] The output of failed pipelines is saved into the Home directory, not the project to [Crunch] All pipeline outputs should go to the parent project, regardless of success or failure
Sally,
The behavior I'm seeing right now is that logs always go to the right project, but outputs only go to the right project when the pipeline instance is successful. This roughly squares with what I see in the code, too. Do you have an example of a pipeline instance where the logs went to the home project?
Updated by Brett Smith almost 10 years ago
crunch-job always sets the log's owner_uuid to the job's owner_uuid, which would seem to be what we want.
It could do the same when it creates the output collection. Right now a-r-p-i makes the new collection in the target project when the component finishes successfully.
Updated by Sarah Guthrie almost 10 years ago
My apologies, you are correct: the logs always go to the right project, but the output collections do not. Having the output collections go into the correct project as well (for both arvados-cwl-runner and the normal crunch runner) would be helpful.
Updated by Brett Smith almost 10 years ago
We want these collections to say "(Failed)" at the end of their name, so the user has a clear indication of that.
This could be done either by moving the relevant block in a-r-p-i down below the case statement it's currently in; or removing that completely, and adding the logic to create_output_collection() in crunch-job. The latter seems a little conceptually cleaner, but maybe more work? Discuss.
Updated by Peter Amstutz