Arvados

I did not see any signs of distress in the nginx or ws logs around the time of the failing test. I believe these are the only two services involved since the nginx test configuration includes a stanza to upgrade websocket connections.

The websockets library we use has had upgrades around this same time. Jenkins currently has 15.0.1. That was true under Debian 12 as well, so that's probably not it.

https://github.com/python-websockets/websockets/issues/1648

https://github.com/python/cpython/issues/137583

but that bisects the problem to Python commit e047a35 which was authored July 2025 so wouldn't be included in the Python 3.10 we're using.

I tried upgrading websockets to 16.0 (the latest version) just because that was easy.

In general, we should pin to a major version anyway. The library does make API breaking changes between major versions. See the changelog

But this didn't help. In fact I'm seeing new problems. First is the same TimeoutError from WebsocketTest.test_subscribe_websocket—but it might just be coincidence we haven't seen that before. Also got this once:

========================================================== FAILURES ==========================================================
_________________________________ WebsocketTest.test_websocket_reconnect_on_unexpected_close _________________________________

self = <tests.test_events.WebsocketTest testMethod=test_websocket_reconnect_on_unexpected_close>

    def test_websocket_reconnect_on_unexpected_close(self):
>       self._test_websocket_reconnect(True)

tests/test_events.py:284: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_events.py:262: in _test_websocket_reconnect
    event = events.get(True, 5)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <queue.Queue object at 0x7f43d7238a90>, block = True, timeout = 5

    def get(self, block=True, timeout=None):
        '''Remove and return an item from the queue.

        If optional args 'block' is true and 'timeout' is None (the default),
        block if necessary until an item is available. If 'timeout' is
        a non-negative number, it blocks at most 'timeout' seconds and raises
        the Empty exception if no item was available within that time.
        Otherwise ('block' is false), return an item if one is immediately
        available, else raise the Empty exception ('timeout' is ignored
        in that case).
        '''
        with self.not_empty:
            if not block:
                if not self._qsize():
                    raise Empty
            elif timeout is None:
                while not self._qsize():
                    self.not_empty.wait()
            elif timeout < 0:
                raise ValueError("'timeout' must be a non-negative number")
            else:
                endtime = time() + timeout
                while not self._qsize():
                    remaining = endtime - time()
                    if remaining <= 0.0:
>                       raise Empty
E                       _queue.Empty

/opt/Python-3.10.19/lib/python3.10/queue.py:179: Empty
----------------------------------------------------- Captured log call ------------------------------------------------------
WARNING  arvados.events:events.py:226 Unexpected close. Reconnecting.
================================================== short test summary info ===================================================
FAILED tests/test_events.py::WebsocketTest::test_websocket_reconnect_on_unexpected_close - _queue.Empty
=============================================== 1 failed, 19 passed in 43.65s ================================================

I still had my Debian 12 VM lying around so I tried on that. Same version of Python (3.10.19) and websockets (15.0.1), run out of main, fresh test temp dir, 45 runs, zero failures. I can do more runs but my initial take is this means it's very unlikely the issue is anything in the Arvados code base or written in Python.

I'm really hoping nginx is the issue at this point, because if it's an underlying C library that's going to be a giant pain to track down and deal with.