Arvados

Part of this story should be a user listing integration test that uses a similar arvados-boot setup to the new federation test.

A related error was reported:

error updating local user records: request failed: http://localhost:8000/arvados/v1/users/batch_update: 422 Unprocessable Entity: #<ActiveRecord::RecordInvalid: Validation failed: Modified by client uuid 'arlog-ozdt8-8bsvojxdycrr5qe' not found> (req-54bqf3xe8p5y1sppyu4u)

modified_by_client_uuid is a special field (I believe it is the client_uuid from token that last touched the record) and it doesn't really make sense to try and copy it to the local users table, because the client_uuid is cluster specific.

Updates at 8ba91a5558a9fbb743c726a7e38f7e7dcd7bf8ee - branch 16263-logincluster-user-list-fix
Test run: developer-run-tests: #1781

• Adds a hook to create and update calls so that every object can declare a list of fields that should be saved as nil when receiving an empty value.
• Adds the same behavior to batch_update on users.
• Adds tests.

Updates at 9fb4398f2
Test run: developer-run-tests: #1782

• Removes modified_by_*_uuid fields to the user's cache field whitelist.

Updates at 6004ef018
Test run: developer-run-tests-remainder: #1855

• Fixes go formatting.

Lucas Di Pentima wrote:

Updates at 9fb4398f2
Test run: developer-run-tests: #1782

• Removes modified_by_*_uuid fields to the user's cache field whitelist.

I noticed 'uuid' isn't in the whitelist. Does it get added in somewhere else?

Peter Amstutz wrote:

Lucas Di Pentima wrote:

Updates at 9fb4398f2
Test run: developer-run-tests: #1782

• Removes modified_by_*_uuid fields to the user's cache field whitelist.

I noticed 'uuid' isn't in the whitelist. Does it get added in somewhere else?

Actually I see now, the uuid is a dictionary key.

I'm running the federation-migrate test (which tests the LoginCluster feature). It's getting an API error:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.6:8000/arvados/v1/users?alt=json&limit=1&filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D returned "invalid filter operand %!q(bool=true)">

Here's the error from controller (I think it fails in controller, it doesn't make it to the API server):

@400000005e7a57190e6981ac.s:2020-03-24_18:50:52.75547 {"PID":40,"RequestID":"req-5otj2n35rtxdh6x5bir3","level":"info","msg":"response","remoteAddr":"127.0.0.1:47752","reqBytes":0,"reqForwardedFor":"172.17.0.9","reqHost":"172.17.0.6:8000","reqMethod":"GET","reqPath":"arvados/v1/users","reqQuery":"alt=json\u0026limit=1\u0026filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D","respBody":"{\"errors\":[\"invalid filter operand %!q(bool=true)\"]}\n","respBytes":53,"respStatus":"Internal Server Error","respStatusCode":500,"time":"2020-03-24T18:50:52.755416243Z","timeToStatus":0.000604,"timeTotal":0.000614,"timeWriteBody":0.000010}

This is the API call from Python:

existing = api.users().list(filters=[["email", "=", "test@example.com"],
                                     ["is_active", "=", True]], limit=1).execute()

Peter Amstutz wrote:

I'm running the federation-migrate test (which tests the LoginCluster feature). It's getting an API error:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.6:8000/arvados/v1/users?alt=json&limit=1&filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D returned "invalid filter operand %!q(bool=true)">

Fixed at 5c941aa10b7b4bf9d0804f98eb93a8104bfb7658 (rebased branch)
Test run: developer-run-tests: #1785

The problem was that the GoSDK wasn't supporting boolean operands on filters when unmarshalling from JSON. Added some tests.

Now I'm getting this:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.8:8000/arvados/v1/users?offset=0&limit=9223372036854775807&alt=json returned "request failed: https://172.17.0.6:8000/arvados/v1/users?cluster_id=&count=&limit=9223372036854776000&offset=0: 500 Internal Server Error: strconv.ParseInt: parsing "9223372036854776000": value out of range">

This is on this SDK call:

ul = arvados.util.list_all(arv.users().list)

The above problem is related to the fact that json.Unmarshal() uses the float64 type to decode JSON numbers, and when decoding really big numbers, it loses precision. See https://stackoverflow.com/questions/16946306/preserve-int64-values-when-parsing-json-in-go

Fix with tests at 12b534152
Test run: developer-run-tests: #1786

I think changing controller's requestAndDecode() behavior is a bit overkill, I'm reading a bit about json's Unmarshaler interface to see if something more specific can be implemented: https://golang.org/pkg/encoding/json/#Unmarshaler

Got another error while trying run the federation migrate test. I'm looking into this one.

(case3@test) Error getting user info for x31je-tpzed-kjnjvyl0m4n1n2i from x31je: <HttpError 401 when requesting https://172.17.0.4:8000/arvados/v1/users/current?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=&remote=&uuid=: 401 Unauthorized: Not logged in (req-y6de37eqyvwzh16wbs37)">

(case3@test) No user listed with same email to migrate xsdok-tpzed-0w3rz8pyzrgryxr to x31je, will create new user with username 'case3'
(case3@test) Error getting user info for x31je-tpzed-o5zmv7q6i8cj5pq from x31je: <HttpError 401 when requesting https://172.17.0.4:8000/arvados/v1/users/current?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=&remote=&uuid=: 401 Unauthorized: Not logged in (req-mfgs9p4efyof1honpt26)">
(case3@test) Migrating xsdok-tpzed-0w3rz8pyzrgryxr to x31je-tpzed-o5zmv7q6i8cj5pq on xqcb8
(case3@test) Error migrating user: <HttpError 422 when requesting https://172.17.0.6:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-0w3rz8pyzrgryxr&new_user_uuid=x31je-tpzed-o5zmv7q6i8cj5pq&new_owner_uuid=xqcb8-j7d0g-9m0sjeteyv2v88y&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 422 Unprocessable Entity: request failed: http://localhost:8004/arvados/v1/users/merge: 422 Unprocessable Entity: User in new_user_uuid not found (req-ghtmi4tlfenkpp1u1ub4)">
(case3@test) Migrating xsdok-tpzed-0w3rz8pyzrgryxr to x31je-tpzed-o5zmv7q6i8cj5pq on xsdok
(case3@test) Updating username of x31je-tpzed-o5zmv7q6i8cj5pq to 'case3' on xsdok

2020-03-30_15:04:25.82187 {"PID":9470,"RequestID":"req-mfgs9p4efyof1honpt26","level":"info","msg":"request","remoteAddr":"127.0.0.1:35268","reqBytes":0,"reqForwardedFor":"172.17.0.1","reqHost":"172.17.0.4:8000","reqMethod":"GET","reqPath":"arvados/v1/users/current","reqQuery":"alt=json","time":"2020-03-30T15:04:25.821786163Z"}

2020-03-30_15:04:25.83428 {"PID":9470,"RequestID":"req-mfgs9p4efyof1honpt26","level":"info","msg":"response","remoteAddr":"127.0.0.1:35268","reqBytes":0,"reqForwardedFor":"172.17.0.1","reqHost":"172.17.0.4:8000","reqMethod":"GET","reqPath":"arvados/v1/users/current","reqQuery":"alt=json","respBody":"{\"errors\":[\"request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=\\u0026remote=\\u0026uuid=: 401 Unauthorized: Not logged in (req-mfgs9p4efyof1honpt26)\"]}\n","respBytes":177,"respStatus":"Unauthorized","respStatusCode":401,"time":"2020-03-30T15:04:25.834223741Z","timeToStatus":0.012422,"timeTotal":0.012436,"timeWriteBody":0.000013}

ESC[1mESC[36mApiClientAuthorization Load (0.4ms)ESC[0m ESC[1mESC[34mSELECT "api_client_authorizations".* FROM "api
_client_authorizations" WHERE (uuid='x31je-gj3su-9zgjsz5tpyw1ewx' and (expires_at is null or expires_at > CURRENT_TIME
STAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1ESC[0m "LIMIT", 1
ESC[1mESC[36mUser Load (0.6ms)ESC[0m ESC[1mESC[34mSELECT "users".* FROM "users" WHERE (redirect_to_user_uuid is nul
l) AND "users"."id" = 13ESC[0m
ESC[1mESC[36mApiClient Load (0.5ms)ESC[0m ESC[1mESC[34mSELECT "api_clients".* FROM "api_clients" WHERE "api_clients
"."id" = 1ESC[0m
[req-mfgs9p4efyof1honpt26] Error 1585580665+9a33fe5f: 401 {"method":"GET","path":"/arvados/v1/users/current","format":"html","controller":"Arvados::V1::UsersController","action
":"current","status":401,"duration":0.87,"view":0.29,"db":0.0,"request_id":"req-mfgs9p4efyof1honpt26","client_ipaddr":
"127.0.0.1","client_auth":null,"params":{"forwarded_for":"","remote":"","uuid":""},"@timestamp":"2020-03-30T15:04:25.8
33481839Z","@version":"1","message":"[401] GET /arvados/v1/users/current (Arvados::V1::UsersController#current)"}

Fixed previous error by adding omitempty to fields of GetOptions.

Next thing failing:

(case3@test) No user listed with same email to migrate xsdok-tpzed-5wtzbkn5iw66h8g to x31je, will create new user with username 'case3'
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on x31je
(case3@test) Error migrating user: <HttpError 403 when requesting https://172.17.0.4:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-5wtzbkn5iw66h8g&new_user_uuid=x31je-tpzed-2ss1g6pqjxhcg9d&new_owner_uuid=x31je-j7d0g-njks8t4fmc96e5r&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 403 Forbidden: request failed: http://localhost:8004/arvados/v1/users/merge: 403 Forbidden: cannot move objects into supplied new_owner_uuid: new user does not have write permission (req-qwbqtchw2ubsk3a57g41)">
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on xqcb8
(case3@test) Error migrating user: <HttpError 403 when requesting https://172.17.0.6:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-5wtzbkn5iw66h8g&new_user_uuid=x31je-tpzed-2ss1g6pqjxhcg9d&new_owner_uuid=xqcb8-j7d0g-wr33pehajc1edr8&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 403 Forbidden: request failed: http://localhost:8004/arvados/v1/users/merge: 403 Forbidden: cannot move objects into supplied new_owner_uuid: new user does not have write permission (req-mxz2gnky6zk8lswzw5em)">
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on xsdok
(case3@test) Updating username of x31je-tpzed-2ss1g6pqjxhcg9d to 'case3' on xsdok

Fixed the error (it was trying to migrate users that didn't exist and didn't need to be migrated, fixed the existence check).

Next error:

(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je
(case7@test) Error updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je: <HttpError 422 when requesting https://172.17.0.4:8000/arvados/v1/users/x31je-tpzed-89a34lxw5j0v92d?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/x31je-tpzed-89a34lxw5j0v92d: 422 Unprocessable Entity: Username has already been taken (req-10dasoeab3iw7ifi1p9h)">
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on x31je
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on xqcb8
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on xqcb8
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on xsdok
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on xsdok

16263-logincluster-user-list-fix @ 5efed8f05e4616ed423124544e5b021e2c8d5a6f

developer-run-tests: #1795

• Adds bypass_federation flag, only usable by admins, to disable federation behavior and get responses only from the local cluster database
• arv-federation-migrate uses bypass_federation flag.
• passes federation migrate test and federated workflow tests. the federation migrate test also tests federated user listing.
• Bumped API revision. arv-federation-migrate checks for new API revision.

After lots of struggling, was able to run the federation tests locally, but wasn't able to run the fed migration tests, I think because of OOM issues.

Code looks good, just one comment: bypass_federation param access test is OK, but it doesn’t test the param behavior. It's now possible to make a federation integration test to confirm that the flag works as intended. The federation migrate tests indirectly check this feature but I think a proper specific test would be needed.

Other than that, it LGTM.