Arvados

About step in self.steps cf. https://github.com/common-workflow-language/cwltool/blob/e37134a90ac7a7c18254e30cff16da590b45c6d7/cwltool/workflow.py#L126

Is there any way to find out whether a step is a command line tool or workflow?

For example,

ordereddict([('run', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/ls-wf.cwl'), ('in', [ordereddict([('source', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/subworkflow-ls-wf.cwl#cat/cattxt'), ('id', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/subworkflow-ls-wf.cwl#ls-wf/txt')])]), ('out', ['file:///home/jiayong/Code/work_scripts/cwl/depdency/subworkflow-ls-wf.cwl#ls-wf/wctxt']), ('id', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/subworkflow-ls-wf.cwl#ls-wf'), ('inputs', [ordereddict([('source', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/subworkflow-ls-wf.cwl#cat/cattxt'), ('id', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/subworkflow-ls-wf.cwl#ls-wf/txt'), ('type', 'File'), ('_tool_entry', ordereddict([('type', 'File'), ('id', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/ls-wf.cwl#txt')]))])]), ('outputs', [ordereddict([('type', 'File'), ('outputSource', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/ls-wf.cwl#wc/wctxt'), ('id', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/subworkflow-ls-wf.cwl#ls-wf/wctxt'), ('_tool_entry', ordereddict([('type', 'File'), ('outputSource', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/ls-wf.cwl#wc/wctxt'), ('id', 'file:///home/jiayong/Code/work_scripts/cwl/depdency/ls-wf.cwl#wctxt')]))])])]) 

It's not directly clear whether the above step is a workflow or not. ("outputSource" is an indicator but it's not very direct.)

Jiayong Li wrote:

About step in self.steps cf. https://github.com/common-workflow-language/cwltool/blob/e37134a90ac7a7c18254e30cff16da590b45c6d7/cwltool/workflow.py#L126

Is there any way to find out whether a step is a command line tool or workflow?

For example,
[...]

It's not directly clear whether the above step is a workflow or not. ("outputSource" is an indicator but it's not very direct.)

Possibly an easier way to do this is to use the visit() method

https://github.com/common-workflow-language/cwltool/blob/e37134a90ac7a7c18254e30cff16da590b45c6d7/cwltool/workflow.py#L175

You pass in a function, it will call that function with each tool or subworkflow that occurs in the workflow.

For each workflow, you just need to check for circular references in the steps. This should be straightforward: starting from the input parameters of the workflow, find what steps refer to those parameters, then find the output parameters, then find the steps that refer to the output parameters, and so forth. Each step you visit, you push it on to a "visited" list. If you find that you are re-visiting a step, that means you have found a cycle, and should raise an error.

I tested adding a line before L135 https://github.com/common-workflow-language/cwltool/blob/e37134a90ac7a7c18254e30cff16da590b45c6d7/cwltool/workflow.py#L135

print(self.steps)

My test workflow is as follows
foo-wf.cwl

cwlVersion: v1.1
class: Workflow
requirements:
  SubworkflowFeatureRequirement: {}

inputs:
  txt:
    type: File

outputs:
  wctxt:
    type: File
    outputSource: subworkflow-foo-wf/wctxt

steps:
  cat:
    run: cat.cwl
    in:
      intxt: txt
    out: [cattxt]
  subworkflow-foo-wf:
    run: subworkflow-foo-wf.cwl
    in:
      txt: cat/cattxt
    out: [wctxt]

subworkflow-foo-wf.cwl

cwlVersion: v1.1
class: Workflow

inputs:
  txt:
    type: File

outputs:
  wctxt:
    type: File
    outputSource: wc/wctxt

steps:
  cat:
    run: cat.cwl
    in:
      intxt: txt
    out: [cattxt]
  ls:
    run: ls.cwl
    in:
      intxt: cat/cattxt
    out: [lstxt]
  wc:
    run: wc.cwl
    in:
      intxt: ls/lstxt
    out: [wctxt]

Running this workflow with the added print(self.steps) yields

[<cwltool.workflow.WorkflowStep object at 0x7fa6afa07390>, <cwltool.workflow.WorkflowStep object at 0x7fa6aa2d16a0>, <cwltool.workflow.WorkflowStep object at 0x7fa6aa7206a0>]
[<cwltool.workflow.WorkflowStep object at 0x7fa6aa253a58>, <cwltool.workflow.WorkflowStep object at 0x7fa6aa209048>]

This means print(self.steps) is called twice, once for subworkflow-foo-wf and once for foo-wf. My questions: are both of these two print commands run before executing the workflows? If we run circular dependency check on subworkflow-foo-wf, then execute it, and then run check on foo-wf and execute, this would defeat the purpose of check before run.

The static checks happen at loading time, not at execution time, so they happen before running, at this point it is only loading the description.

Just need to update the cwltool version used by a-c-r to get the new check.