Bug #2800: [Crunch] Remove global state in Python SDK - Arvados

Actions

Copy link

Bug #2800

closed

[Crunch] Remove global state in Python SDK

Added by Tom Clegg almost 11 years ago. Updated over 10 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Brett Smith

Category:

Target version:

2014-08-27 Sprint

Start date:

08/18/2014

Due date:

% Done:

100%

Estimated time:

(Total: 3.00 h)

Story points:

1.0

Description

It should be easy for a Python program to communicate with API and Keep storage servers at multiple sites in one process. For example:

a = arvados.api('v1', host='localhost', token='12345', insecure=True)
b = arvados.api('v1', host='qr1hi.arvadosapi.com', token='xyzzy')

x = a.collections().list().execute()

w1 = CollectionWriter(client=a)
w2 = CollectionWriter(client=b)
w1.write('foo')
w2.write('bar')

Subtasks 2 (0 open — 2 closed)

Actions

Copy link

Updated by Tom Clegg almost 11 years ago

Target version set to Arvados Future Sprints

Actions

Copy link

Updated by Tom Clegg over 10 years ago

Description updated (diff)

Actions

Copy link

Updated by Tom Clegg over 10 years ago

Status changed from New to In Progress
Assigned To set to Tom Clegg

Actions

Copy link

Updated by Tom Clegg over 10 years ago

Started 2800-python-global-state branch.

Actions

Copy link

Updated by Tom Clegg over 10 years ago

On 2800-pgs branch at c5f8a89

Interface

Accept host, token, and insecure arguments in the api() call. If provided, use them rather than looking up environment variables.
- This change obsoletes some awkward maneuvers with environment variables (some of this removed from tests/test_api.py).

Caching

arvados.api() no longer returns a cached connection object with different credentials, endpoint, or "insecure" flag than the connection being requested. (Caching code used to assume only the requested version could change between calls to api().)
- Example: this change makes it unnecessary for tests/test_api.py to clear the connection cache to prevent accidental future use of its fake credentials. Removed.
If the caller says cache=False, don't use an existing connection (as before), but don't put the newly created connection object in the cache either.

Token handling

The credentials supplied to (or looked up by) arvados.api() are used for the life of the connection it returns. Previously, CredentialsFromEnv looked up the auth token in the global os.environ during each API method call.

Mocking

test_api.py uses a mock for API requests but still needs a discovery document. Now it uses run_test_server.py instead of relying on qr1hi.arvadosapi.com. This makes the test suite slower but more meaningful.

Actions

Copy link

Updated by Tom Clegg over 10 years ago

Target version changed from Arvados Future Sprints to 2014-08-27 Sprint

(The issue of using Keep services at multiple sites (global_client_object in keep.py) hasn't been addressed yet.)

Actions

Copy link

Updated by Brett Smith over 10 years ago

Just a few small comments on c5f8a89

We can skip the step of hashing strings for a cache key. We can just create a tuple of all the values we care about, and use that directly as the lookup key. What do you think of that approach?
Please keep imports sorted alphabetically.
Please retain the comment in test_api.py about needing a real discovery document, to explain why we're running a test server.

Thanks.

Actions

Copy link

Updated by Tom Clegg over 10 years ago

Brett Smith wrote:

We can skip the step of hashing strings for a cache key. We can just create a tuple of all the values we care about, and use that directly as the lookup key. What do you think of that approach?

Ah, yes, I was limiting myself to string keys for no reason. Now using tuple as dictionary key.