Arvados

Copied from #10028:

I tried some queries on test servers and found

• counting lots of rows in a huge table is slow, regardless of how great the index is: in order to count accurately, postgresql has to visit every counted row
• given this, postgresql might be outsmarting us when it does a seq scan in our test trials. It knows most of the rows match the query, which means it will have to visit them anyway when it does a count -- so there is not much benefit in consulting the index first.

Typically, the huge table that causes trouble is "logs". For example:

• arv-mount: subscribe to all events

         self.events = arvados.events.subscribe(self._api_client,
                                   [["event_type", "in", ["create", "update", "delete"]]],
                                   self.on_event)
  

• Python SDK:

  items = self.api.logs().list(limit=1, order="id desc", filters=f).execute()['items']

• API server:

       list[:items_available] = @objects.
          except(:limit).except(:offset).
          count(:id, distinct: true)
  

• Postgresql: Slowly count all rows of the giant logs table with event_type ∈ {"create", "update", "delete"}.

Nobody here actually cares about items_available.

All PollClient needs is max(id) from the logs table, and only for the purpose of passing it to the next logs().list() call.

Perhaps it could pass a filter like [["created_at",">=",subscription_time]] (instead of doing this expensive API call to find the last known ID and then passing [["logs.id",">",str(self.id)]]) ...?

10224-efficient-event-poll-startup

test 0e98d5f2f9c827deb5beb4a1765a4718d7cffd88

0e98d5f

API unit tests failed (log_test)

One python test failed: test_subscribe_poll (tests.test_events.WebsocketTest)

I didn’t run the fuse tests

With the tests fixed, LGTM

API unit tests fixed.

Python SDK tests pass for me. We'll see what Jenkins thinks -- build 56 queued at https://ci.curoverse.com/job/developer-run-tests/