Bug #2800
closed
[Crunch] Remove global state in Python SDK
Added by Tom Clegg almost 11 years ago.
Updated over 10 years ago.
Estimated time:
(Total: 3.00 h)
Description
It should be easy for a Python program to communicate with API and Keep storage servers at multiple sites in one process. For example:
a = arvados.api('v1', host='localhost', token='12345', insecure=True)
b = arvados.api('v1', host='qr1hi.arvadosapi.com', token='xyzzy')
x = a.collections().list().execute()
w1 = CollectionWriter(client=a)
w2 = CollectionWriter(client=b)
w1.write('foo')
w2.write('bar')
- Target version set to Arvados Future Sprints
- Description updated (diff)
- Status changed from New to In Progress
- Assigned To set to Tom Clegg
Started 2800-python-global-state branch.
On 2800-pgs branch at c5f8a89
Interface
- Accept host, token, and insecure arguments in the
api() call. If provided, use them rather than looking up environment variables.
- This change obsoletes some awkward maneuvers with environment variables (some of this removed from
tests/test_api.py).
Caching
arvados.api() no longer returns a cached connection object with different credentials, endpoint, or "insecure" flag than the connection being requested. (Caching code used to assume only the requested version could change between calls to api().)
- Example: this change makes it unnecessary for
tests/test_api.py to clear the connection cache to prevent accidental future use of its fake credentials. Removed.
- If the caller says
cache=False, don't use an existing connection (as before), but don't put the newly created connection object in the cache either.
Token handling
- The credentials supplied to (or looked up by)
arvados.api() are used for the life of the connection it returns. Previously, CredentialsFromEnv looked up the auth token in the global os.environ during each API method call.
Mocking
test_api.py uses a mock for API requests but still needs a discovery document. Now it uses run_test_server.py instead of relying on qr1hi.arvadosapi.com. This makes the test suite slower but more meaningful.
- Target version changed from Arvados Future Sprints to 2014-08-27 Sprint
(The issue of using Keep services at multiple sites (global_client_object in keep.py) hasn't been addressed yet.)
Just a few small comments on c5f8a89
- We can skip the step of hashing strings for a cache key. We can just create a tuple of all the values we care about, and use that directly as the lookup key. What do you think of that approach?
- Please keep imports sorted alphabetically.
- Please retain the comment in
test_api.py about needing a real discovery document, to explain why we're running a test server.
Thanks.
Brett Smith wrote:
- We can skip the step of hashing strings for a cache key. We can just create a tuple of all the values we care about, and use that directly as the lookup key. What do you think of that approach?
Ah, yes, I was limiting myself to string keys for no reason. Now using tuple as dictionary key.
- Please keep imports sorted alphabetically.
Fixed (by not importing hashlib)
- Please retain the comment in
test_api.py about needing a real discovery document, to explain why we're running a test server.
Indeed. Restored.
Thanks... now at 1e36870
Tom Clegg wrote:
Brett Smith wrote:
- Please keep imports sorted alphabetically.
Fixed (by not importing hashlib)
This is still an issue in test_api.py. Everything else looks good, so please go ahead and merge with that change. Thanks.
- Assigned To changed from Tom Clegg to Brett Smith
2800-pysdk-no-global-keep-client-wip is up for review to deprecate the global Keep client.
- Status changed from In Progress to Resolved
- % Done changed from 33 to 100
Applied in changeset arvados|commit:05bea2c50474edeb9d0e3fb8daaf838b58ea9a54.
- Subject changed from Remove global state in Python SDK to [Crunch] Remove global state in Python SDK
Also available in: Atom
PDF
Updated by 
Updated by 
Added by Brett Smith over 10 years ago
Merge branch '2800-pgs' refs #2800