Arvados

I wrote a workflow that took a File[] input and mapped that to command line tool arguments (busybox's cat -V if it matters). Then I ran the workflow where the input was a single S3 file URL.

arvados-cwl-runner --defer-downloads --submit first logged:

WARNING Download error: <HttpError 422 when requesting https://z2a01:8443/arvados/v1/credentials?filters=%5B%5B%22credential_class%22%2C+%22%3D%22%2C+%22aws_access_key%22%5D%2C+%5B%22scopes%22%2C+%22%3D%22%2C+%5B%5D%5D%2C+%5B%22expires_at%22%2C+%22%3E%22%2C+%222025-10-13T21%3A01%3A06.072122Z%22%5D%5D&alt=json returned "#<ArgumentError: Invalid operand type 'Array' for '=' operator in filters> (req-hrgyzzm1le0zsvxgms4b)">

As best I can tell this is just an incorrect filter in resolve_aws_key in source:sdk/cwl/arvados_cwl/pathmapper.py. But the workflow was submitted. It has these logs:

INFO /opt/arvados-py/bin/arvados-cwl-runner 3.2.0.dev20251002205250, arvados-python-client 3.2.0.dev20251001190205, cwltool 3.1.20240508115724
INFO Resolved '/var/lib/cwl/workflow.json#main' to 'file:///var/lib/cwl/workflow.json#main'
INFO Using cluster z2a01 (https://z2a01/)
INFO Running inside container z2a01-dz642-65rte5mvi85h6ei
WARNING Download error: boto3 did not find any local AWS credentials to use to download from S3. If you want to use credentials registered with Arvados, use --defer-downloads. If the bucket is public, use --s3-public-bucket.
ERROR Workflow error, try again with --debug for more information:
Can't handle 's3://test-curii-nocreds/aws-s3-bulk-download.cwl'
Container exited with status code 1

Why is it telling me to use --defer-downloads when I already did? Because the logic in ArvPathMapper.visit in source:sdk/cwl/arvados_cwl/pathmapper.py doesn't actually know whether it's running as a container via --submit or not, all it knows is that it didn't find credentials and therefore it tells you to use this switch in the hopes that it'll find them in Arvados.

If I go ahead and add --debug I get stack traces but they're not any more helpful. Here's the local end:

WARNING Download error: <HttpError 422 when requesting https://z2a01:8443/arvados/v1/credentials?filters=%5B%5B%22credential_class%22%2C+%22%3D%22%2C+%22aws_access_key%22%5D%2C+%5B%22scopes%22%2C+%22%3D%22%2C+%5B%5D%5D%2C+%5B%22expires_at%22%2C+%22%3E%22%2C+%222025-10-13T21%3A07%3A34.481253Z%22%5D%5D&alt=json returned "#<ArgumentError: Invalid operand type 'Array' for '=' operator in filters> (req-sdg8rs4pyyenemf11qo1)">
Traceback (most recent call last):
  File "/home/brett/venv22996/lib/python3.11/site-packages/arvados_cwl/pathmapper.py", line 177, in visit
    self.arvrunner.selected_credential = resolve_aws_key(self.arvrunner.api, src)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brett/venv22996/lib/python3.11/site-packages/arvados_cwl/pathmapper.py", line 63, in resolve_aws_key
    ["expires_at", ">", expires_at]]).execute()
                                      ^^^^^^^^^
  File "/home/brett/venv22996/lib/python3.11/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brett/venv22996/lib/python3.11/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
arvados.errors.ApiError: <HttpError 422 when requesting https://z2a01:8443/arvados/v1/credentials?filters=%5B%5B%22credential_class%22%2C+%22%3D%22%2C+%22aws_access_key%22%5D%2C+%5B%22scopes%22%2C+%22%3D%22%2C+%5B%5D%5D%2C+%5B%22expires_at%22%2C+%22%3E%22%2C+%222025-10-13T21%3A07%3A34.481253Z%22%5D%5D&alt=json returned "#<ArgumentError: Invalid operand type 'Array' for '=' operator in filters> (req-sdg8rs4pyyenemf11qo1)">

And in the container logs:

WARNING Download error: boto3 did not find any local AWS credentials to use to download from S3. If you want to use credentials registered with Arvados, use --defer-downloads. If the bucket is public, use --s3-public-bucket.
Traceback (most recent call last):
File "/opt/arvados-py/lib/python3.11/site-packages/arvados_cwl/pathmapper.py", line 168, in visit
raise WorkflowException("boto3 did not find any local AWS credentials to use to download from S3. If you want to use credentials registered with Arvados, use --defer-downloads. If the bucket is public, use --s3-public-bucket.")
cwltool.errors.WorkflowException: boto3 did not find any local AWS credentials to use to download from S3. If you want to use credentials registered with Arvados, use --defer-downloads. If the bucket is public, use --s3-public-bucket.
DEBUG EXIT mapper 1760389361.394021 2.0970864295959473
DEBUG EXIT upload_job_order 1760389361.3940828 2.0984396934509277
ERROR Workflow error:
Can't handle 's3://test-curii-nocreds/aws-s3-bulk-download.cwl'
Traceback (most recent call last):
File "/opt/arvados-py/lib/python3.11/site-packages/cwltool/main.py", line 1314, in main
(out, status) = real_executor(
^^^^^^^^^^^^^^
File "/opt/arvados-py/lib/python3.11/site-packages/arvados_cwl/executor.py", line 714, in arv_executor
job_order, jobmapper = upload_job_order(self, "%s input" % runtimeContext.name,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/arvados-py/lib/python3.11/site-packages/arvados_cwl/runner.py", line 708, in upload_job_order
jobmapper = upload_dependencies(arvrunner,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/arvados-py/lib/python3.11/site-packages/arvados_cwl/runner.py", line 432, in upload_dependencies
mapper = ArvPathMapper(arvrunner, sc, "",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/arvados-py/lib/python3.11/site-packages/arvados_cwl/pathmapper.py", line 89, in __init__
super(ArvPathMapper, self).__init__(referenced_files, input_basedir, None)
File "/opt/arvados-py/lib/python3.11/site-packages/cwltool/pathmapper.py", line 104, in __init__
self.setup(dedup(referenced_files), basedir)
File "/opt/arvados-py/lib/python3.11/site-packages/arvados_cwl/pathmapper.py", line 326, in setup
self.addentry(srcobj, c, ".", remap)
File "/opt/arvados-py/lib/python3.11/site-packages/arvados_cwl/pathmapper.py", line 226, in addentry
raise SourceLine(obj, "location", WorkflowException).makeError("Can't handle '%s'" % obj["location"])
cwltool.errors.WorkflowException: Can't handle 's3://test-curii-nocreds/aws-s3-bulk-download.cwl'
Container exited with status code 1

This is a bad experience because the error messages never tell you what the real problem is, let alone clearly. I think the flow should be this:

1. As soon as you run with S3 inputs, a-c-r should resolve credentials and confirm they work.
   1. First check if it's a public bucket. If so, no credentials needed.
   2. Otherwise, if using --enable-s3-credentials-capture, try the local credentials.
   3. Otherwise, search Arvados for credentials and try those.
      • If any credentials are going to expire within the next 10 days, warn the user about that.
   4. If nothing worked, fail immediately with an error.
2. If you're deferring downloads, submit the a-c-r container to Arvados with secret_mounts that specify the credentials to use.
   1. When credentials are available this way, a-c-r just uses them no questions asked. They're expected to work and if they don't it's an error. It doesn't search again or anything like that.
   2. Exactly how a-c-r finds these credentials and how they're formatted TBD.
      • It would be nice if we could write them in a format that boto3 could load directly so we don't have to write translation code.
      • Maybe the container request can do that and set environment variables like AWS_CREDENTIALS_SECRET_FILE to tell boto3 where it is? That'd be neat.