Arvados

I would also like to point out that I will predict the empty source information, provenance graphs, used-by graphs, and even pipeline instance graphs will also appear if one arv-copies a pipeline instance between projects with different permissions. Preventing this would be nice.

Sarah Guthrie wrote:

I would also like to point out that I will predict the empty source information, provenance graphs, used-by graphs, and even pipeline instance graphs will also appear if one arv-copies a pipeline instance between projects with different permissions. Preventing this would be nice.

Yeah, I'm pretty sure you're right. I agree that it's worth fixing the arv-copy bug at its source too, and that's what I want this ticket to represent. But figuring out the mechanics of copying jobs is going to be a little more involved.

• The "log" links point to the log tab of the "show job" page for the job, even though the job UUID doesn't exist here. Perhaps this can be fixed by making a route to the "show job" page like "component X from pipeline instance Y" (this would also make the job page more context-aware, e.g., it could show the "component name" of the job in the relevant pipeline, and it could show the pipeline in breadcrumbs).
• The graph tab (I'm guessing) tries to look up the latest job records, skips each job because that doesn't work, and violates some assumption that there will be at least one component to display. If we make it fall back on the pipeline instance record instead, it should have everything needed to make a graph.

Tom Clegg wrote:

• The graph tab (I'm guessing) tries to look up the latest job records, skips each job because that doesn't work, and violates some assumption that there will be at least one component to display. If we make it fall back on the pipeline instance record instead, it should have everything needed to make a graph.

The basic crash is #7008. We should consider extending the fix there like you describe.

But I'm not sure that fix would address all our graph problems? For example, I don't think that suffices to render provenance graphs for collections that come out of the pipeline instance. To do that, we need to be able to figure out which jobs the collection came from, and we can't query pipeline instance components this way.

Brett Smith wrote:

But I'm not sure that fix would address all our graph problems? For example, I don't think that suffices to render provenance graphs for collections that come out of the pipeline instance. To do that, we need to be able to figure out which jobs the collection came from, and we can't query pipeline instance components this way.

It's true, this wouldn't be enough to get foreign jobs to appear in the "which jobs lead to this collection?" graphs even if pipelines have been copied from the foreign cluster -- but this part sounds as much like a feature as a bug.

It also doesn't help resolve the discrepancy between job and pipeline sharing when those items are in different projects (e.g., you can see the provenance graph on the pipeline page, but if you ask "which jobs lead to {pipeline output}" you get nothing). This part doesn't sound much like a feature.

One way to look at this is that we need weaker job/output claims (weaker than "this arvados system ran this job as described here") to be visible to users, without conflating those weak claims with the strong claims in the jobs table. The distinction between a real job and an arbitrary user-created job record (from arv-copy or running crunch-job in local/dev mode) is rather hard for a user to see, might not even be taken into account when reusing jobs, etc. One possibility is to fix this (with, essentially, a "real" flag) such that we can copy jobs all over the place without worrying about them getting inadvertently re-queued, edited by hand and then erroneously used by automatic job reuse, etc.

Tom Clegg wrote:

Brett Smith wrote:

But I'm not sure that fix would address all our graph problems? For example, I don't think that suffices to render provenance graphs for collections that come out of the pipeline instance. To do that, we need to be able to figure out which jobs the collection came from, and we can't query pipeline instance components this way.

It's true, this wouldn't be enough to get foreign jobs to appear in the "which jobs lead to this collection?" graphs even if pipelines have been copied from the foreign cluster -- but this part sounds as much like a feature as a bug.

I get what you're saying that the cluster doesn't have much assurance that the provenance actually is what's recorded in the pipeline instance, but having disagreeing provenance graphs doesn't seem like the most helpful way to communicate that. I think ideally the graph would be rendered differently, and maybe include a footnote explaining that the results may not be reflected across the cluster.

It also doesn't help resolve the discrepancy between job and pipeline sharing when those items are in different projects (e.g., you can see the provenance graph on the pipeline page, but if you ask "which jobs lead to {pipeline output}" you get nothing). This part doesn't sound much like a feature.

FWIW, I think that's an equally important application, and maybe more pressing to fix since it's easier. It's become a common pipeline development idiom to work on a pipeline in a project, then after you get a clean, successful run, copy that instance to a new project to share and show off. In order for that to be effective, all the associated metadata needs to come along to the new project.