Bug #11693
closed
[Crunch2] Correctly handle symlinks to input files in output dir
100%
Description
Crunch v1 and cwltool allow the output directory to have symlinks to input files, which are then efficiently propagated to output. Unfortunately, crunch-run fails in this case. Some scripts rely on this behavior. It needs to support dereferencing symlinks and adding them to the output collection.
Updated by Lucas Di Pentima over 7 years ago
Reviewed a340487a7
Did a jenkins test run because my local tests kept failing for some reason, all ok: https://ci.curoverse.com/job/developer-run-tests/286/
- File
services/crunch-run/crunchrun.go:962:outputSuffixalready assigned onL952 - What should happen when the symlink is pointing to a mount, but that mount is not a collection? (asking because of the
...&& mnt.Kind == “collection”bit onL958). Can a mount be other thing than a collection? Shouldn't the collection mounts be filtered when building thebindslist onL924?
Updated by Peter Amstutz over 7 years ago
Lucas Di Pentima wrote:
Reviewed a340487a7
Just a minor comment and a couple of questions:
Did a jenkins test run because my local tests kept failing for some reason, all ok: https://ci.curoverse.com/job/developer-run-tests/286/
- File
services/crunch-run/crunchrun.go:962:outputSuffixalready assigned onL952
Copy and paste error. Fixed.
- What should happen when the symlink is pointing to a mount, but that mount is not a collection? (asking because of the
...&& mnt.Kind == “collection”bit onL958). Can a mount be other thing than a collection? Shouldn't the collection mounts be filtered when building thebindslist onL924?
Inputs are always collections. Symlinks to files in temporary directories isn't supported.
Filtering the binds array to just collections is a good idea. Fixed.
Updated by Peter Amstutz over 7 years ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|commit:f26f70d0a60798065c5f7a5cb91b95587cc9e9ef.