Arvados

h1. Routing multi cluster requests

h2. Concept

The goal of federation is to present an interface that fuses multiple clusters into a single view.

This requires a router or proxy which determines which cluster(s) a request should go.  This could exist in several places: entirely in the client, in a dedicated request router service, on the local cluster's API server.

For the purposes of this discussion, we'll consider how this works when implemented entirely on the client side.  A router service is likely to have similar request handling behavior but has the additional requirement to act as a transparent API server proxy.

h2. Examples

My "home cluster" is qr1hi.  I have a token qr1hi-secretsecretsecret.

h3. I want to read a collection on c97qk using the Python SDK.

<pre>

c = CollectionReader("c97qk-...")

</pre>

# The CollectionReader calls the request router class.

# The request router class examines the prefix c97qk and contacts c97qk.arvadosapi.com.

# The request router uses the "salted" token hmac(c97qk, qr1hi-secretsecretsecret) → qr1hi-secretsecretc97qk

# c97qk gets the token and notices the qr1hi prefix.

# c97qk contacts qr1hi to determine if the token is valid and what user is associated with the token.

# c97qk caches the token and sets current_user.  The request proceeds as normal.

# The request gets the response and returns it to CollectionReader.

#* manifest_text needs to be munged by either the c97qk or the request router to add the hint "+K@c97qk" so that blocks will be fetched from c97qk.

h3. I want to search for a collection across clusters

<pre>

c = router.collections().list(filters=[["name", "like", "sample-1234%"]]).execute()

</pre>

# The router has a "search list" of clusters (where does this come from??? maybe an attribute of the primary user account  on qr1hi?)

# The router sends the request to each cluster in parallel using federated identity / salted token described above.

# The router gathers the results.

# The router collates the results (will need to understand "order" option to do this properly)

# Collated results are returned

# Paging - ???  likely need to keep track of some state locally to be able to be able to issue correct follow-up requests to each cluster.  Can have consistent ordering within a page but not across pages unless all pages are fetched first.

Another case: I want to list the contents of a project across clusters.  Same query process.

<pre>

c = router.collections().list(owner_uuid="qr1hi-....").execute()

</pre>

h3. I want to create a collection on another cluster.

Provide "owner_uuid" of a project or group on a foreign cluster.

<pre>

router.collections().create(body={"owner_uuid": "c97qk-...."}).execute()

</pre>

# The request router class examines the prefix c97qk and contacts c97qk.arvadosapi.com using federated identity / salted token described above .

# The cluster determines if the user has write access to the group or project and validates the create request as normal.

# The newly created record is returned.

No "owner_uuid" means creating the object on the "home" cluster.

h3. I want to update an object on another cluster.

<pre>

router.collections().update(uuid="c97qk-....", body={....}).execute()

</pre>

# The request router class examines the prefix c97qk and contacts c97qk.arvadosapi.com using federated identity / salted token described above .

# The cluster determines if the user has write access to object and validates the update request as normal.

# The updated record is returned.

h3. I want to change the ownership of a remote object to a project on my home cluster.

The object is located on c97qk and currently owned by me, I'd like to make it owned by a project qr1hi-...

# Route the "update" as described above to c97qk.

# c97qk contacts qr1hi and asks if the user has write access to the project.

# The object is updated and returned to the user

(This suggests I can only share things with groups on the same home cluster as me.  hmm.)

h3. I want to change the ownership of an object on my home cluster object to a project on a remote cluster.

# Route the "update" as described above to qr1hi.

# qr1hi contacts c97qk _using the c97qk salted token_ and asks if the user has write access to the project.

# The object is updated and returned to the user

h3. I want to change the ownership of an object from one remote project (c97qk) to another (4xphq).

Can't be done directly (???) because c97qk and 4xphq don't talk to each other directly.  (The token given to c97qk is not valid for accessing 4xphq and likewise).  Could be done as a two-step process where ownership is assigned from c97qk to qr1hi, then from qr1hi to 4xphq.