Bug #4048: [Crunch] arv-run-pipeline-instance --run-jobs-here cannot run jobs with local script path in script_version. - Arvados

Subject changed from arv-run-pipeline-instance --run-job-here cannot run jobs with local script path in script_version. to [Crunch] arv-run-pipeline-instance --run-job-here cannot run jobs with local script path in script_version.
Category set to Crunch
Assigned To set to Brett Smith
Target version set to 2015-01-07 sprint

Will verify.

Actions

Copy link

Updated by Brett Smith about 10 years ago

Subject changed from [Crunch] arv-run-pipeline-instance --run-job-here cannot run jobs with local script path in script_version. to [Crunch] arv-run-pipeline-instance --run-jobs-here cannot run jobs with local script path in script_version.
Assigned To deleted (~~Brett Smith~~)
Target version changed from 2015-01-07 sprint to Bug Triage

Nope:

$ arv-run-pipeline-instance --run-jobs-here --template grep_hash_pipeline.json  c1_grep::input=c1bad4b39ca5a924e481008009d94e32+210
arv-run-pipeline-instance 15598: create job: ["script_version filter needs repository filter"] with attributes {:job=>{:script=>"grep", :script_parameters=>{:input=>"c1bad4b39ca5a924e481008009d94e32+210", :pattern=>"a"}, :script_version=>"/home/brett/4048/", :repository=>"", :owner_uuid=>"qr1hi-tpzed-5jakibnrp1qpty1", :is_locked_by_uuid=>"qr1hi-tpzed-5jakibnrp1qpty1", :submit_id=>"instance qr1hi-d1hrv-n93q0alr04p80k0 rand 2x9u9ba3nc62j", :state=>"Running"}, :find_or_create=>true}
arv-run-pipeline-instance 15598: component c1_grep new job failed

2014-12-10 22:37:51 +0000 -- pipeline_instance qr1hi-d1hrv-n93q0alr04p80k0
c1_grep -                           -
c2_hash -                           -
arv-run-pipeline-instance 15598: job creation failed - giving up on this pipeline instance

This is happening partly because of the new-ish filters support in the job/create API method, and the way that arv-run-pipeline-instance creates those filters from particular job parameters. I think arv-run-pipeline-instance should not send filters, or try to reuse jobs at all, when it's running local jobs.

Actions

Copy link

Updated by Brett Smith about 10 years ago

Since it's apparently relevant: my pipeline template has the script directory listed in script_version, and an empty repository field, following the example in our documentation.

Actions

Copy link

Updated by Brett Smith about 10 years ago

If I make a local git repository, and try to run with that as repository, it doesn't work either.

Relevant job definition from the template:

        "c1_grep": {
            "script": "grep",
            "script_parameters": {
                "input": {
                    "required": true,
                    "dataclass": "Collection" 
                },
                "pattern": "a" 
            },
            "repository": "/home/brett/4048",
            "script_version": "master" 
        },

Running:

brett@shell.qr1hi:~/4048$ pwd
/home/brett/4048
brett@shell.qr1hi:~/4048$ git rev-list master
5cd5e8502f8213d17965f23c044ef3ee224e1cf2
brett@shell.qr1hi:~/4048$ arv-run-pipeline-instance --run-jobs-here --template grep_hash_pipeline.json  c1_grep::input=c1bad4b39ca5a924e481008009d94e32+210
arv-run-pipeline-instance 3334: create job: ["error searching /home/brett/4048 from 'master' to 'master', excluding []"] with attributes {:job=>{:script=>"grep", :script_parameters=>{:input=>"c1bad4b39ca5a924e481008009d94e32+210", :pattern=>"a"}, :script_version=>"master", :repository=>"/home/brett/4048", :owner_uuid=>"qr1hi-tpzed-5jakibnrp1qpty1", :is_locked_by_uuid=>"qr1hi-tpzed-5jakibnrp1qpty1", :submit_id=>"instance qr1hi-d1hrv-hvn98gb0pujulmy rand 1bjmrbav6cdnh", :state=>"Running"}, :find_or_create=>true}
arv-run-pipeline-instance 3334: component c1_grep new job failed

2014-12-11 15:23:33 +0000 -- pipeline_instance qr1hi-d1hrv-hvn98gb0pujulmy
c1_grep -                           -
c2_hash -                           -
arv-run-pipeline-instance 3334: job creation failed - giving up on this pipeline instance

Actions

Copy link