Arvados

Category:

Target version:

2018-09-05 Sprint

Start date:

08/15/2018

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Story points:

3.0

Release:

Arvados 1.2

Release relationship:

Auto

Description

To get "projects shared with me" efficiently requires a new API endpoint. The way arv-mount currently determines what to list in "shared" currently requires looking at all projects and finding the ones where owner_uuid is not another project which is visible to us (meaning: users, non-project groups, or shared subprojects where the parent is not visible). This is expensive to compute on
the client, but can probably be accomplished with a single query on the API server.

Proposed endpoint: /arvados/v1/groups/shared

Return in "items" a list of toplevel projects and collections which are shared with me. Toplevel projects are ones which are either owned by a user, owned by a non-project group, or owned by a group which the user cannot read.

Support "includes" to return the set of owner objects as well (users, groups).

Possible query (abbreviated):

select groups that are readable by current user AND
- the owner_uuid is a user (not a group) OR
- the owner_uuid is a group is not a project or is not readable by the current user

select uuid from groups where 
  exists(select 1 from materialized_permission_view where user_uuid='2tlax-tpzed-51vcbmac4uv6bpb' and target_uuid=groups.uuid) and 
  (groups.owner_uuid in (select uuid from users) or 
    not exists(select 1 from materialized_permission_view where user_uuid='2tlax-tpzed-51vcbmac4uv6bpb' and target_uuid=groups.owner_uuid)) and 
  owner_uuid != '2tlax-tpzed-51vcbmac4uv6bpb';

Subtasks 2 (0 open — 2 closed)

Task #13950: Review 13146-shared-rails

Resolved

08/16/2018

Task #14060: shared endpoint

Resolved

08/15/2018

Related issues 5 (2 open — 3 closed)

Related to Arvados - Bug #12990: [FUSE] Access shared/ is inefficient

In Progress

Related to Arvados - Story #13111: [WebDAV] Support browsing of project hierarchies

Resolved

Tom Clegg

03/27/2018

Related to Arvados - Story #13218: Support browsing of projects shared with me in WebDAV

New

Related to Arvados - Feature #14201: [API] Accept "exclude_home_project" flag in groups#contents

Resolved

10/01/2018

Related to Arvados Workbench 2 - Support #15238: [Workbench] Change the content inside the owner column from uuid to Projects and user names

Resolved

Issue # Delay: days Cancel

History
Notes
Property changes
Associated revisions

Actions

Updated by Peter Amstutz almost 7 years ago

Status changed from New to In Progress

Actions

Updated by Peter Amstutz almost 7 years ago

Description updated (diff)

Actions

Updated by Peter Amstutz almost 7 years ago

Related to Bug #12990: [FUSE] Access shared/ is inefficient added

Actions

Updated by Peter Amstutz almost 7 years ago

Related to Story #13111: [WebDAV] Support browsing of project hierarchies added

Actions

Updated by Tom Morris almost 7 years ago

Tracker changed from Feature to Story
Status changed from In Progress to New

Actions

Updated by Tom Morris almost 7 years ago

Related to Story #13218: Support browsing of projects shared with me in WebDAV added

Actions

Updated by Peter Amstutz over 6 years ago

Description updated (diff)

Actions

Updated by Peter Amstutz over 6 years ago

Description updated (diff)

The current algorithm used in the FUSE driver is:

get all projects
go through the list of projects and find projects which have an owner_uuid that isn't in the list and isn't the current user. This has the effect of finding the projects for which the owner is either a user, a group which is not a project, or a group which is not readable by the current user.
The list of "roots" consists of each owner (if the owner is readable but not a project) or the project itself (if not).

Actions

Updated by Peter Amstutz over 6 years ago

Description updated (diff)
Target version changed from To Be Groomed to Arvados Future Sprints
Story points set to 3.0

Actions

#10

Updated by Tom Morris over 6 years ago

Target version changed from Arvados Future Sprints to 2018-08-15 Sprint

Actions

#11

Updated by Peter Amstutz over 6 years ago

Assigned To set to Peter Amstutz
Target version changed from 2018-08-15 Sprint to Arvados Future Sprints

Actions

#12

Updated by Peter Amstutz over 6 years ago

Target version changed from Arvados Future Sprints to 2018-08-15 Sprint

Actions

#13

Updated by Peter Amstutz over 6 years ago

Status changed from New to In Progress

Actions

#14

Updated by Peter Amstutz over 6 years ago

Target version changed from 2018-08-15 Sprint to 2018-09-05 Sprint

Actions

https://ci.curoverse.com/view/Developer/job/developer-run-tests/850/

#15

Updated by Peter Amstutz over 6 years ago

13146-shared-rails @ 0ba8a53ba145475153b01dc498f85dbd2f03228b

I spent a while mulling over alternate APIs for "shared" but kept coming back the original definition. It is a bit ugly but it should still be much more efficient than listing every project and user and working it out on the client side.

Actions

#16

Updated by Tom Clegg over 6 years ago

Suggestions

rename the "include" response field to "included" like http://jsonapi.org/format/#document-compound-documents
...and rename "includes" to "included" in the comment in shared()
skip the owners unless the client asks for them with ?include=owner like http://jsonapi.org/format/#fetching-includes
apply @select to the related objects as well

Have you tried these queries on a production-size database to make sure performance is OK with the existing indices?

Actions

https://ci.curoverse.com/view/Developer/job/developer-run-tests/855/

#18

Updated by Peter Amstutz over 6 years ago

Tom Clegg wrote:

Suggestions

rename the "include" response field to "included" like http://jsonapi.org/format/#document-compound-documents

...and rename "includes" to "included" in the comment in shared()

skip the owners unless the client asks for them with ?include=owner like http://jsonapi.org/format/#fetching-includes>

apply @select to the related objects as well

Done.

Have you tried these queries on a production-size database to make sure performance is OK with the existing indices?

See previous note.

Now 13146-shared-rails @ a41c0f6aa41b658c8f2947c46cb90778894f5cf3

Actions

#19

Updated by Tom Clegg over 6 years ago

Experiments & include/included changes LGTM.

In the docs...

The second paragraph should have some reference to the "include" flag, perhaps just the word "optionally" -- as written it implies "always".
It feels like we're saying the same thing 4 different ways, including some pseudocode that has "a and b or c or d", which makes me think about operator precedence, and an "intended use" paragraph that seems to amount to "this API exists to support clients which wish to do what this API does". Could we pare this back to one or two ways of explaining it?

Actions

#20

Updated by Peter Amstutz over 6 years ago

Tom Clegg wrote:

Experiments & include/included changes LGTM.
In the docs...

The second paragraph should have some reference to the "include" flag, perhaps just the word "optionally" -- as written it implies "always".

It feels like we're saying the same thing 4 different ways, including some pseudocode that has "a and b or c or d", which makes me think about operator precedence, and an "intended use" paragraph that seems to amount to "this API exists to support clients which wish to do what this API does". Could we pare this back to one or two ways of explaining it?

Sure, documentation updated @ 489aed58cb0d8bd816e07128cfcb9f5a06224083

Actions

#21

Updated by Tom Clegg over 6 years ago

LGTM, thanks.

(Nit: seems like "this is useful for clients which wish to browse the list of" could be replaced with "i.e.," since it's implicit that an API is useful for clients that wish to get the information it returns... but this is gold-plated enough and I'm ready to move on)

Actions

#22

Updated by Tom Morris over 6 years ago

It looks like a branch got merged a few days ago, but this ticket is still marked as being "In Progress." Is there additional work to be done? We were looking at it during the Workbench2 review to see what it's status was.

Actions

#23

Updated by Peter Amstutz over 6 years ago

Status changed from In Progress to Resolved

Tom Morris wrote:

It looks like a branch got merged a few days ago, but this ticket is still marked as being "In Progress." Is there additional work to be done? We were looking at it during the Workbench2 review to see what it's status was.

This is done. The workbench2 folks should be able to use it (provided they are accessing one of the dev clusters with the latest code).

Actions