Actions
Bug #13550
closed
nearly all postgres connections blocked by row-level locks on containers
Status:
Closed
Priority:
Normal
Assigned To:
-
Category:
API
Target version:
-
Story points:
-
Description
When heavily loaded, our API server database gets locked up with nearly all of its connections waiting on the exact same row-level lock for the exact same query:
postgres=# select * from pg_stat_activity where state = 'active' and waiting = 't';
datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | state_change | waiting | state | backend_xid | backend_xmin | query
-------+------------------------+-------+----------+---------+-----------------------------------------------------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+---------+--------+-------------+--------------+----------------------------------------------------------------------------------------------------
16385 | arvados_api_production | 24604 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 34064 | 2018-05-31 00:49:18.909929+00 | 2018-05-31 01:14:09.11903+00 | 2018-05-31 01:14:09.647224+00 | 2018-05-31 01:14:09.647225+00 | t | active | 439006197 | 438993530 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24287 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 59982 | 2018-05-31 00:41:07.435114+00 | 2018-05-31 01:15:36.186618+00 | 2018-05-31 01:15:36.685339+00 | 2018-05-31 01:15:36.68534+00 | t | active | 439006613 | 439000502 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24288 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 60002 | 2018-05-31 00:41:07.542624+00 | 2018-05-31 01:14:47.890354+00 | 2018-05-31 01:14:48.353241+00 | 2018-05-31 01:14:48.353241+00 | t | active | 439006540 | 438997392 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24290 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 60100 | 2018-05-31 00:41:07.753147+00 | 2018-05-31 01:17:07.098924+00 | 2018-05-31 01:17:07.55296+00 | 2018-05-31 01:17:07.552961+00 | t | active | 439007724 | 439002877 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24291 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 60144 | 2018-05-31 00:41:07.865569+00 | 2018-05-31 01:14:33.098273+00 | 2018-05-31 01:14:33.580657+00 | 2018-05-31 01:14:33.580657+00 | t | active | 439006274 | 438996997 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24293 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 60278 | 2018-05-31 00:41:08.103182+00 | 2018-05-31 01:10:10.896482+00 | 2018-05-31 01:10:11.372779+00 | 2018-05-31 01:10:11.372779+00 | t | active | 439004359 | 438979273 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24295 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 60348 | 2018-05-31 00:41:08.320945+00 | 2018-05-31 01:13:36.938558+00 | 2018-05-31 01:13:37.37965+00 | 2018-05-31 01:13:37.37965+00 | t | active | 439006128 | 438991155 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24304 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61076 | 2018-05-31 00:41:09.433588+00 | 2018-05-31 01:12:49.124591+00 | 2018-05-31 01:12:49.57235+00 | 2018-05-31 01:12:49.572351+00 | t | active | 439005948 | 438986380 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24305 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61178 | 2018-05-31 00:41:09.580259+00 | 2018-05-31 01:12:19.489636+00 | 2018-05-31 01:12:19.949881+00 | 2018-05-31 01:12:19.949881+00 | t | active | 439005464 | 438983871 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24306 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61262 | 2018-05-31 00:41:09.695598+00 | 2018-05-31 01:17:57.223021+00 | 2018-05-31 01:17:57.670667+00 | 2018-05-31 01:17:57.670668+00 | t | active | 439007961 | 439004074 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24307 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61370 | 2018-05-31 00:41:09.816626+00 | 2018-05-31 01:17:51.533152+00 | 2018-05-31 01:17:51.997229+00 | 2018-05-31 01:17:51.997229+00 | t | active | 439007952 | 439004041 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24308 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61462 | 2018-05-31 00:41:09.9303+00 | 2018-05-31 01:15:56.185524+00 | 2018-05-31 01:15:56.645485+00 | 2018-05-31 01:15:56.645486+00 | t | active | 439006880 | 439000801 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24309 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61526 | 2018-05-31 00:41:10.066496+00 | 2018-05-31 01:12:36.736683+00 | 2018-05-31 01:12:37.21644+00 | 2018-05-31 01:12:37.216441+00 | t | active | 439005500 | 438984494 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24310 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61592 | 2018-05-31 00:41:10.186742+00 | 2018-05-31 01:14:09.768611+00 | 2018-05-31 01:14:10.236008+00 | 2018-05-31 01:14:10.236009+00 | t | active | 439006223 | 438993531 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24311 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61652 | 2018-05-31 00:41:10.298494+00 | 2018-05-31 01:09:44.648776+00 | 2018-05-31 01:09:45.121644+00 | 2018-05-31 01:09:45.121644+00 | t | active | 439004303 | 438975382 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24322 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61732 | 2018-05-31 00:41:10.425406+00 | 2018-05-31 01:12:33.910817+00 | 2018-05-31 01:12:34.39455+00 | 2018-05-31 01:12:34.39455+00 | t | active | 439005497 | 438984215 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24323 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61770 | 2018-05-31 00:41:10.54604+00 | 2018-05-31 01:12:54.149685+00 | 2018-05-31 01:12:54.609289+00 | 2018-05-31 01:12:54.60929+00 | t | active | 439005956 | 438986401 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24327 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61876 | 2018-05-31 00:41:11.022763+00 | 2018-05-31 01:16:57.416142+00 | 2018-05-31 01:16:57.890549+00 | 2018-05-31 01:16:57.89055+00 | t | active | 439007490 | 439002776 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24329 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61926 | 2018-05-31 00:41:11.23925+00 | 2018-05-31 01:10:50.105786+00 | 2018-05-31 01:10:50.584818+00 | 2018-05-31 01:10:50.584819+00 | t | active | 439004444 | 438980141 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24331 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61946 | 2018-05-31 00:41:11.354912+00 | 2018-05-31 01:12:22.294551+00 | 2018-05-31 01:12:22.76993+00 | 2018-05-31 01:12:22.769931+00 | t | active | 439005467 | 438983935 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24332 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 61978 | 2018-05-31 00:41:11.469755+00 | 2018-05-31 01:16:35.235726+00 | 2018-05-31 01:16:35.713786+00 | 2018-05-31 01:16:35.713786+00 | t | active | 439007405 | 439002514 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24333 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62046 | 2018-05-31 00:41:11.626576+00 | 2018-05-31 01:13:02.790714+00 | 2018-05-31 01:13:03.280366+00 | 2018-05-31 01:13:03.280367+00 | t | active | 439005974 | 438987449 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24335 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62138 | 2018-05-31 00:41:11.904313+00 | 2018-05-31 01:14:03.19896+00 | 2018-05-31 01:14:03.660498+00 | 2018-05-31 01:14:03.660498+00 | t | active | 439006165 | 438992821 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24337 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62210 | 2018-05-31 00:41:12.142955+00 | 2018-05-31 01:13:57.317233+00 | 2018-05-31 01:13:57.781923+00 | 2018-05-31 01:13:57.781923+00 | t | active | 439006154 | 438991352 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24338 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62276 | 2018-05-31 00:41:12.287509+00 | 2018-05-31 01:15:22.01192+00 | 2018-05-31 01:15:22.495284+00 | 2018-05-31 01:15:22.495285+00 | t | active | 439006595 | 439000412 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24339 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62394 | 2018-05-31 00:41:12.411755+00 | 2018-05-31 01:14:33.364808+00 | 2018-05-31 01:14:33.820132+00 | 2018-05-31 01:14:33.820133+00 | t | active | 439006276 | 438996997 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24340 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62534 | 2018-05-31 00:41:12.552716+00 | 2018-05-31 01:16:07.358854+00 | 2018-05-31 01:16:07.817971+00 | 2018-05-31 01:16:07.817971+00 | t | active | 439007352 | 439001079 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24341 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62656 | 2018-05-31 00:41:12.683598+00 | 2018-05-31 01:14:06.136795+00 | 2018-05-31 01:14:06.609656+00 | 2018-05-31 01:14:06.609657+00 | t | active | 439006170 | 438992824 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24342 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62714 | 2018-05-31 00:41:12.81789+00 | 2018-05-31 01:17:23.375245+00 | 2018-05-31 01:17:23.841879+00 | 2018-05-31 01:17:23.84188+00 | t | active | 439007908 | 439003770 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24345 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62766 | 2018-05-31 00:41:13.161326+00 | 2018-05-31 01:13:51.454385+00 | 2018-05-31 01:13:51.95407+00 | 2018-05-31 01:13:51.95407+00 | t | active | 439006148 | 438991284 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24346 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62786 | 2018-05-31 00:41:13.274506+00 | 2018-05-31 01:10:19.337269+00 | 2018-05-31 01:10:19.8202+00 | 2018-05-31 01:10:19.820201+00 | t | active | 439004371 | 438979495 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24347 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62800 | 2018-05-31 00:41:13.395824+00 | 2018-05-31 01:15:01.960458+00 | 2018-05-31 01:15:02.427972+00 | 2018-05-31 01:15:02.427973+00 | t | active | 439006565 | 438998751 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24348 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62814 | 2018-05-31 00:41:13.523133+00 | 2018-05-31 01:10:25.058239+00 | 2018-05-31 01:10:25.521572+00 | 2018-05-31 01:10:25.521573+00 | t | active | 439004380 | 438979710 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24350 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62898 | 2018-05-31 00:41:13.778136+00 | 2018-05-31 01:17:11.698512+00 | 2018-05-31 01:17:12.158526+00 | 2018-05-31 01:17:12.158527+00 | t | active | 439007879 | 439002998 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24351 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62914 | 2018-05-31 00:41:13.915238+00 | 2018-05-31 01:13:48.547248+00 | 2018-05-31 01:13:49.012363+00 | 2018-05-31 01:13:49.012363+00 | t | active | 439006141 | 438991244 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24352 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62940 | 2018-05-31 00:41:14.048318+00 | 2018-05-31 01:09:56.483482+00 | 2018-05-31 01:09:56.948463+00 | 2018-05-31 01:09:56.948464+00 | t | active | 439004332 | 438976868 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24353 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62960 | 2018-05-31 00:41:14.169252+00 | 2018-05-31 01:14:09.786907+00 | 2018-05-31 01:14:10.30827+00 | 2018-05-31 01:14:10.308271+00 | t | active | 439006224 | 438993531 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24355 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63004 | 2018-05-31 00:41:14.443383+00 | 2018-05-31 01:14:19.070205+00 | 2018-05-31 01:14:19.535458+00 | 2018-05-31 01:14:19.535458+00 | t | active | 439006252 | 438994364 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24356 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63046 | 2018-05-31 00:41:14.559165+00 | 2018-05-31 01:17:11.551966+00 | 2018-05-31 01:17:12.015414+00 | 2018-05-31 01:17:12.015414+00 | t | active | 439007878 | 439002998 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24357 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63068 | 2018-05-31 00:41:14.673343+00 | 2018-05-31 01:14:59.122974+00 | 2018-05-31 01:14:59.589426+00 | 2018-05-31 01:14:59.589427+00 | t | active | 439006561 | 438998547 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24358 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63110 | 2018-05-31 00:41:14.805165+00 | 2018-05-31 01:15:41.779584+00 | 2018-05-31 01:15:42.234742+00 | 2018-05-31 01:15:42.234743+00 | t | active | 439006646 | 439000594 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24362 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63434 | 2018-05-31 00:41:15.396979+00 | 2018-05-31 01:14:24.527603+00 | 2018-05-31 01:14:25.006539+00 | 2018-05-31 01:14:25.00654+00 | t | active | 439006258 | 438994391 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24363 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63530 | 2018-05-31 00:41:15.583457+00 | 2018-05-31 01:09:59.649138+00 | 2018-05-31 01:10:00.091963+00 | 2018-05-31 01:10:00.091964+00 | t | active | 439004344 | 438978912 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24364 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63632 | 2018-05-31 00:41:15.755436+00 | 2018-05-31 01:10:30.626818+00 | 2018-05-31 01:10:31.082076+00 | 2018-05-31 01:10:31.082077+00 | t | active | 439004391 | 438979785 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24368 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63830 | 2018-05-31 00:41:16.46559+00 | 2018-05-31 01:15:10.330529+00 | 2018-05-31 01:15:10.773629+00 | 2018-05-31 01:15:10.773629+00 | t | active | 439006576 | 439000371 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24370 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64090 | 2018-05-31 00:41:16.845548+00 | 2018-05-31 01:14:03.436794+00 | 2018-05-31 01:14:03.903176+00 | 2018-05-31 01:14:03.903176+00 | t | active | 439006167 | 438992821 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24371 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64212 | 2018-05-31 00:41:17.072537+00 | 2018-05-31 01:12:57.018603+00 | 2018-05-31 01:12:57.463083+00 | 2018-05-31 01:12:57.463084+00 | t | active | 439005964 | 438986648 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24372 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64330 | 2018-05-31 00:41:17.273703+00 | 2018-05-31 01:14:12.44826+00 | 2018-05-31 01:14:12.918345+00 | 2018-05-31 01:14:12.918346+00 | t | active | 439006232 | 438993577 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24373 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64446 | 2018-05-31 00:41:17.487986+00 | 2018-05-31 01:15:13.320738+00 | 2018-05-31 01:15:13.796223+00 | 2018-05-31 01:15:13.796223+00 | t | active | 439006580 | 439000372 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24374 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64568 | 2018-05-31 00:41:17.672758+00 | 2018-05-31 01:17:17.760105+00 | 2018-05-31 01:17:18.218807+00 | 2018-05-31 01:17:18.218807+00 | t | active | 439007899 | 439003103 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24375 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64736 | 2018-05-31 00:41:17.809902+00 | 2018-05-31 01:14:36.28183+00 | 2018-05-31 01:14:36.734789+00 | 2018-05-31 01:14:36.73479+00 | t | active | 439006282 | 438997092 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24376 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64928 | 2018-05-31 00:41:18.00934+00 | 2018-05-31 01:12:48.308161+00 | 2018-05-31 01:12:48.77162+00 | 2018-05-31 01:12:48.771621+00 | t | active | 439005941 | 438986380 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24378 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 65112 | 2018-05-31 00:41:18.368481+00 | 2018-05-31 01:14:12.602294+00 | 2018-05-31 01:14:13.075093+00 | 2018-05-31 01:14:13.075093+00 | t | active | 439006233 | 438993577 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24379 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 65196 | 2018-05-31 00:41:18.520142+00 | 2018-05-31 01:14:39.238261+00 | 2018-05-31 01:14:39.705067+00 | 2018-05-31 01:14:39.705067+00 | t | active | 439006285 | 438997101 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24380 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 65334 | 2018-05-31 00:41:18.691754+00 | 2018-05-31 01:13:54.583611+00 | 2018-05-31 01:13:55.055204+00 | 2018-05-31 01:13:55.055205+00 | t | active | 439006151 | 438991290 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24381 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 65432 | 2018-05-31 00:41:18.834567+00 | 2018-05-31 01:11:17.781977+00 | 2018-05-31 01:11:18.243432+00 | 2018-05-31 01:11:18.243433+00 | t | active | 439004717 | 438980435 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24382 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 65532 | 2018-05-31 00:41:18.995834+00 | 2018-05-31 01:15:04.818893+00 | 2018-05-31 01:15:05.273041+00 | 2018-05-31 01:15:05.273042+00 | t | active | 439006568 | 438998754 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24383 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 1114 | 2018-05-31 00:41:19.153664+00 | 2018-05-31 01:11:20.658353+00 | 2018-05-31 01:11:21.124754+00 | 2018-05-31 01:11:21.124755+00 | t | active | 439004720 | 438980437 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24384 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 1170 | 2018-05-31 00:41:19.286778+00 | 2018-05-31 01:11:46.106113+00 | 2018-05-31 01:11:46.542477+00 | 2018-05-31 01:11:46.542478+00 | t | active | 439004801 | 438983054 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24387 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 1376 | 2018-05-31 00:41:19.569418+00 | 2018-05-31 01:09:59.493812+00 | 2018-05-31 01:09:59.939374+00 | 2018-05-31 01:09:59.939375+00 | t | active | 439004343 | 438978912 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24388 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 1494 | 2018-05-31 00:41:19.743627+00 | 2018-05-31 01:15:19.096129+00 | 2018-05-31 01:15:19.561881+00 | 2018-05-31 01:15:19.561882+00 | t | active | 439006591 | 439000376 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24391 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 1860 | 2018-05-31 00:41:20.353787+00 | 2018-05-31 01:13:34.047086+00 | 2018-05-31 01:13:34.505358+00 | 2018-05-31 01:13:34.505359+00 | t | active | 439006120 | 438990913 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24392 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2258 | 2018-05-31 00:41:20.834005+00 | 2018-05-31 01:11:43.27159+00 | 2018-05-31 01:11:43.791287+00 | 2018-05-31 01:11:43.791287+00 | t | active | 439004778 | 438980935 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24394 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2270 | 2018-05-31 00:41:20.839454+00 | 2018-05-31 01:15:10.491389+00 | 2018-05-31 01:15:10.954537+00 | 2018-05-31 01:15:10.954538+00 | t | active | 439006577 | 439000371 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24395 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2278 | 2018-05-31 00:41:20.843522+00 | 2018-05-31 01:17:37.871446+00 | 2018-05-31 01:17:38.361759+00 | 2018-05-31 01:17:38.361759+00 | t | active | 439007936 | 439003810 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24396 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2460 | 2018-05-31 00:41:21.004277+00 | 2018-05-31 01:14:12.731913+00 | 2018-05-31 01:14:13.204457+00 | 2018-05-31 01:14:13.204457+00 | t | active | 439006234 | 438993577 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24397 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2590 | 2018-05-31 00:41:21.203469+00 | 2018-05-31 01:12:48.183416+00 | 2018-05-31 01:12:48.638541+00 | 2018-05-31 01:12:48.638542+00 | t | active | 439005940 | 438986380 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24399 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2660 | 2018-05-31 00:41:21.359015+00 | 2018-05-31 01:16:35.094063+00 | 2018-05-31 01:16:35.561679+00 | 2018-05-31 01:16:35.561679+00 | t | active | 439007404 | 439002514 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24400 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2778 | 2018-05-31 00
:41:21.503034+00 | 2018-05-31 01:11:48.793166+00 | 2018-05-31 01:11:49.256705+00 | 2018-05-31 01:11:49.256705+00 | t | active | 439004804 | 438983157 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24401 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2958 | 2018-05-31 00:41:21.702541+00 | 2018-05-31 01:12:42.402287+00 | 2018-05-31 01:12:42.851176+00 | 2018-05-31 01:12:42.851177+00 | t | active | 439005701 | 438985223 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24402 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3042 | 2018-05-31 00:41:21.904585+00 | 2018-05-31 01:17:46.018523+00 | 2018-05-31 01:17:46.475247+00 | 2018-05-31 01:17:46.475248+00 | t | active | 439007945 | 439004033 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24403 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3138 | 2018-05-31 00:41:22.04974+00 | 2018-05-31 01:12:49.259476+00 | 2018-05-31 01:12:49.708451+00 | 2018-05-31 01:12:49.708452+00 | t | active | 439005949 | 438986380 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24405 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3340 | 2018-05-31 00:41:22.418649+00 | 2018-05-31 01:13:31.503771+00 | 2018-05-31 01:13:31.970178+00 | 2018-05-31 01:13:31.970179+00 | t | active | 439006031 | 438990877 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24406 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3456 | 2018-05-31 00:41:22.566214+00 | 2018-05-31 01:17:14.793473+00 | 2018-05-31 01:17:15.26849+00 | 2018-05-31 01:17:15.26849+00 | t | active | 439007894 | 439003083 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24408 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3664 | 2018-05-31 00:41:22.946599+00 | 2018-05-31 01:14:18.690268+00 | 2018-05-31 01:14:19.154219+00 | 2018-05-31 01:14:19.15422+00 | t | active | 439006249 | 438994364 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24409 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3810 | 2018-05-31 00:41:23.1463+00 | 2018-05-31 01:17:14.503255+00 | 2018-05-31 01:17:15.025948+00 | 2018-05-31 01:17:15.025948+00 | t | active | 439007891 | 439003082 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24413 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 4320 | 2018-05-31 00:41:23.962014+00 | 2018-05-31 01:15:47.255315+00 | 2018-05-31 01:15:47.715996+00 | 2018-05-31 01:15:47.715997+00 | t | active | 439006654 | 439000627 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24414 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 4416 | 2018-05-31 00:41:24.177913+00 | 2018-05-31 01:13:34.100298+00 | 2018-05-31 01:13:34.564032+00 | 2018-05-31 01:13:34.564032+00 | t | active | 439006121 | 438990913 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24415 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 4600 | 2018-05-31 00:41:24.489497+00 | 2018-05-31 01:10:33.497809+00 | 2018-05-31 01:10:33.962152+00 | 2018-05-31 01:10:33.962152+00 | t | active | 439004413 | 438979850 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24416 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 4850 | 2018-05-31 00:41:24.78093+00 | 2018-05-31 01:13:00.014195+00 | 2018-05-31 01:13:00.463513+00 | 2018-05-31 01:13:00.463513+00 | t | active | 439005971 | 438987256 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24418 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 5166 | 2018-05-31 00:41:25.494468+00 | 2018-05-31 01:17:26.227862+00 | 2018-05-31 01:17:26.690158+00 | 2018-05-31 01:17:26.690159+00 | t | active | 439007914 | 439003773 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24422 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 6112 | 2018-05-31 00:41:26.665102+00 | 2018-05-31 01:14:00.260595+00 | 2018-05-31 01:14:00.708131+00 | 2018-05-31 01:14:00.708131+00 | t | active | 439006157 | 438991398 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24430 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 11608 | 2018-05-31 00
:41:59.199351+00 | 2018-05-31 01:17:15.087924+00 | 2018-05-31 01:17:15.552041+00 | 2018-05-31 01:17:15.552042+00 | t | active | 439007896 | 439003083 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24436 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 11728 | 2018-05-31 00:41:59.725009+00 | 2018-05-31 01:16:51.930309+00 | 2018-05-31 01:16:52.386573+00 | 2018-05-31 01:16:52.386574+00 | t | active | 439007466 | 439002760 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24444 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 12590 | 2018-05-31 00:42:05.038192+00 | 2018-05-31 01:16:35.033643+00 | 2018-05-31 01:16:35.473975+00 | 2018-05-31 01:16:35.473976+00 | t | active | 439007403 | 439002514 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24450 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 13932 | 2018-05-31 00:42:13.843525+00 | 2018-05-31 01:11:14.011495+00 | 2018-05-31 01:11:14.47559+00 | 2018-05-31 01:11:14.475591+00 | t | active | 439004711 | 438980370 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24451 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 14380 | 2018-05-31 00:42:16.70234+00 | 2018-05-31 01:14:09.799158+00 | 2018-05-31 01:14:10.310271+00 | 2018-05-31 01:14:10.310271+00 | t | active | 439006225 | 438993531 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24454 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 14770 | 2018-05-31 00:42:19.633167+00 | 2018-05-31 01:14:50.471454+00 | 2018-05-31 01:14:50.939941+00 | 2018-05-31 01:14:50.939942+00 | t | active | 439006546 | 438997442 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24458 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 15468 | 2018-05-31 00:42:24.335829+00 | 2018-05-31 01:15:19.231917+00 | 2018-05-31 01:15:19.704973+00 | 2018-05-31 01:15:19.704973+00 | t | active | 439006592 | 439000376 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24459 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 15854 | 2018-05-31 00:42:27.284371+00 | 2018-05-31 01:17:12.047917+00 | 2018-05-31 01:17:12.514946+00 | 2018-05-31 01:17:12.514947+00 | t | active | 439007880 | 439002998 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24462 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 16378 | 2018-05-31 00:42:30.616713+00 | 2018-05-31 01:09:50.5421+00 | 2018-05-31 01:09:51.006357+00 | 2018-05-31 01:09:51.006357+00 | t | active | 439004322 | 438976605 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24463 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 16404 | 2018-05-31 00:42:30.808361+00 | 2018-05-31 01:17:34.843652+00 | 2018-05-31 01:17:35.303366+00 | 2018-05-31 01:17:35.303366+00 | t | active | 439007928 | 439003806 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24465 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 16980 | 2018-05-31 00:42:33.683493+00 | 2018-05-31 01:14:08.981728+00 | 2018-05-31 01:14:09.470107+00 | 2018-05-31 01:14:09.470107+00 | t | active | 439006182 | 438993529 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24467 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 17322 | 2018-05-31 00:42:36.447726+00 | 2018-05-31 01:14:56.053732+00 | 2018-05-31 01:14:56.52216+00 | 2018-05-31 01:14:56.522161+00 | t | active | 439006553 | 438997633 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24479 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 19062 | 2018-05-31 00:42:47.166996+00 | 2018-05-31 01:09:50.683351+00 | 2018-05-31 01:09:51.143548+00 | 2018-05-31 01:09:51.143548+00 | t | active | 439004323 | 438976605 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24485 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 22442 | 2018-05-31 00:43:07.957679+00 | 2018-05-31 01:10:02.32029+00 | 2018-05-31 01:10:02.764362+00 | 2018-05-31 01:10:02.764362+00 | t | active | 439004347 | 438978914 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24519 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 28702 | 2018-05-31 00
:44:11.226623+00 | 2018-05-31 01:12:13.790295+00 | 2018-05-31 01:12:14.239483+00 | 2018-05-31 01:12:14.239484+00 | t | active | 439005455 | 438983823 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24520 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 28704 | 2018-05-31 00:44:11.228559+00 | 2018-05-31 01:13:14.316028+00 | 2018-05-31 01:13:14.81503+00 | 2018-05-31 01:13:14.81503+00 | t | active | 439005986 | 438990032 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24524 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 28716 | 2018-05-31 00:44:11.279614+00 | 2018-05-31 01:14:44.320302+00 | 2018-05-31 01:14:44.804543+00 | 2018-05-31 01:14:44.804544+00 | t | active | 439006412 | 438997238 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24611 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 35300 | 2018-05-31 00:49:22.671572+00 | 2018-05-31 01:09:59.365559+00 | 2018-05-31 01:09:59.850502+00 | 2018-05-31 01:09:59.850503+00 | t | active | 439004342 | 438978912 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24615 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 35836 | 2018-05-31 00:49:24.065015+00 | 2018-05-31 01:10:36.138981+00 | 2018-05-31 01:10:36.63693+00 | 2018-05-31 01:10:36.63693+00 | t | active | 439004425 | 438979931 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24619 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 37050 | 2018-05-31 00:49:26.724024+00 | 2018-05-31 01:10:16.438613+00 | 2018-05-31 01:10:16.908346+00 | 2018-05-31 01:10:16.908347+00 | t | active | 439004365 | 438979385 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24629 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 44948 | 2018-05-31 00:49:43.577451+00 | 2018-05-31 01:14:53.480801+00 | 2018-05-31 01:14:53.963048+00 | 2018-05-31 01:14:53.963049+00 | t | active | 439006549 | 438997450 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24633 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52004 | 2018-05-31 00:50:01.946312+00 | 2018-05-31 01:12:45.118595+00 | 2018-05-31 01:12:45.579672+00 | 2018-05-31 01:12:45.579673+00 | t | active | 439005849 | 438985264 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24646 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52074 | 2018-05-31 00:50:01.994861+00 | 2018-05-31 01:16:52.462605+00 | 2018-05-31 01:16:52.928263+00 | 2018-05-31 01:16:52.928264+00 | t | active | 439007471 | 439002760 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24640 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52060 | 2018-05-31 00:50:01.991822+00 | 2018-05-31 01:11:46.077617+00 | 2018-05-31 01:11:46.541419+00 | 2018-05-31 01:11:46.541419+00 | t | active | 439004800 | 438983054 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24638 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52056 | 2018-05-31 00:50:01.990367+00 | 2018-05-31 01:13:45.68724+00 | 2018-05-31 01:13:46.163823+00 | 2018-05-31 01:13:46.163824+00 | t | active | 439006138 | 438991231 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24648 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52112 | 2018-05-31 00:50:02.01516+00 | 2018-05-31 01:15:44.497228+00 | 2018-05-31 01:15:44.972531+00 | 2018-05-31 01:15:44.972532+00 | t | active | 439006649 | 439000619 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24645 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52072 | 2018-05-31 00:50:02.005318+00 | 2018-05-31 01:17:34.654115+00 | 2018-05-31 01:17:35.165501+00 | 2018-05-31 01:17:35.165501+00 | t | active | 439007927 | 439003806 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24651 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52132 | 2018-05-31 00:50:02.030337+00 | 2018-05-31 01:14:03.309564+00 | 2018-05-31 01:14:03.755272+00 | 2018-05-31 01:14:03.755273+00 | t | active | 439006166 | 438992821 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24656 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52146 | 2018-05-31 00
:50:02.035319+00 | 2018-05-31 01:12:11.052038+00 | 2018-05-31 01:12:11.501784+00 | 2018-05-31 01:12:11.501784+00 | t | active | 439005452 | 438983658 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24653 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52136 | 2018-05-31 00:50:02.031769+00 | 2018-05-31 01:18:00.067623+00 | 2018-05-31 01:18:00.511763+00 | 2018-05-31 01:18:00.511763+00 | t | active | 439007964 | 439004127 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24655 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52144 | 2018-05-31 00:50:02.043944+00 | 2018-05-31 01:12:33.032553+00 | 2018-05-31 01:12:33.526084+00 | 2018-05-31 01:12:33.526084+00 | t | active | 439005495 | 438984014 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24661 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52158 | 2018-05-31 00:50:02.042038+00 | 2018-05-31 01:16:01.542364+00 | 2018-05-31 01:16:01.96772+00 | 2018-05-31 01:16:01.96772+00 | t | active | 439007341 | 439000831 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24664 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52168 | 2018-05-31 00:50:02.045116+00 | 2018-05-31 01:13:22.666607+00 | 2018-05-31 01:13:23.129613+00 | 2018-05-31 01:13:23.129613+00 | t | active | 439005995 | 438990228 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24652 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52134 | 2018-05-31 00:50:02.0308+00 | 2018-05-31 01:17:20.510622+00 | 2018-05-31 01:17:20.960233+00 | 2018-05-31 01:17:20.960234+00 | t | active | 439007902 | 439003131 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24677 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52200 | 2018-05-31 00:50:02.060464+00 | 2018-05-31 01:16:21.095214+00 | 2018-05-31 01:16:21.57139+00 | 2018-05-31 01:16:21.571391+00 | t | active | 439007371 | 439002099 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24665 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52170 | 2018-05-31 00:50:02.045721+00 | 2018-05-31 01:12:59.885928+00 | 2018-05-31 01:13:00.356475+00 | 2018-05-31 01:13:00.356476+00 | t | active | 439005970 | 438987256 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24672 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52190 | 2018-05-31 00:50:02.058117+00 | 2018-05-31 01:11:37.440282+00 | 2018-05-31 01:11:37.934608+00 | 2018-05-31 01:11:37.934608+00 | t | active | 439004759 | 438980668 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24667 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52174 | 2018-05-31 00:50:02.047075+00 | 2018-05-31 01:13:17.071201+00 | 2018-05-31 01:13:17.594208+00 | 2018-05-31 01:13:17.594208+00 | t | active | 439005989 | 438990183 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24669 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52180 | 2018-05-31 00:50:02.048519+00 | 2018-05-31 01:14:15.616382+00 | 2018-05-31 01:14:16.091873+00 | 2018-05-31 01:14:16.091874+00 | t | active | 439006237 | 438993636 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24675 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52196 | 2018-05-31 00:50:02.059518+00 | 2018-05-31 01:12:25.100143+00 | 2018-05-31 01:12:25.569932+00 | 2018-05-31 01:12:25.569932+00 | t | active | 439005470 | 438983978 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24679 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52204 | 2018-05-31 00:50:02.062153+00 | 2018-05-31 01:15:54.347347+00 | 2018-05-31 01:15:54.81087+00 | 2018-05-31 01:15:54.810871+00 | t | active | 439006781 | 439000777 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24670 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52182 | 2018-05-31 00:50:02.048999+00 | 2018-05-31 01:17:54.289905+00 | 2018-05-31 01:17:54.792097+00 | 2018-05-31 01:17:54.792097+00 | t | active | 439007955 | 439004066 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24682 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52212 | 2018-05-31 00
:50:02.065993+00 | 2018-05-31 01:12:56.898917+00 | 2018-05-31 01:12:57.363994+00 | 2018-05-31 01:12:57.363994+00 | t | active | 439005963 | 438986648 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24683 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52216 | 2018-05-31 00:50:02.073205+00 | 2018-05-31 01:17:43.278684+00 | 2018-05-31 01:17:43.731574+00 | 2018-05-31 01:17:43.731575+00 | t | active | 439007942 | 439004032 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24687 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52236 | 2018-05-31 00:50:02.107868+00 | 2018-05-31 01:09:36.191151+00 | 2018-05-31 01:09:36.695409+00 | 2018-05-31 01:09:36.69541+00 | t | active | 439004293 | 438974779 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24685 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52226 | 2018-05-31 00:50:02.084404+00 | 2018-05-31 01:15:16.388317+00 | 2018-05-31 01:15:16.831473+00 | 2018-05-31 01:15:16.831474+00 | t | active | 439006586 | 439000374 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24692 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52360 | 2018-05-31 00:50:02.696833+00 | 2018-05-31 01:16:07.225356+00 | 2018-05-31 01:16:07.690581+00 | 2018-05-31 01:16:07.690582+00 | t | active | 439007351 | 439001079 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24694 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52378 | 2018-05-31 00:50:02.728996+00 | 2018-05-31 01:12:05.551298+00 | 2018-05-31 01:12:06.00639+00 | 2018-05-31 01:12:06.00639+00 | t | active | 439005154 | 438983414 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24695 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52420 | 2018-05-31 00:50:02.832425+00 | 2018-05-31 01:17:57.089194+00 | 2018-05-31 01:17:57.554796+00 | 2018-05-31 01:17:57.554796+00 | t | active | 439007960 | 439004074 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24696 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52518 | 2018-05-31 00:50:03.086569+00 | 2018-05-31 01:09:38.988014+00 | 2018-05-31 01:09:39.450279+00 | 2018-05-31 01:09:39.450279+00 | t | active | 439004296 | 438974787 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24697 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52546 | 2018-05-31 00:50:03.189559+00 | 2018-05-31 01:17:28.963462+00 | 2018-05-31 01:17:29.444183+00 | 2018-05-31 01:17:29.444184+00 | t | active | 439007917 | 439003776 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24698 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52628 | 2018-05-31 00:50:03.402487+00 | 2018-05-31 01:15:47.382514+00 | 2018-05-31 01:15:47.844758+00 | 2018-05-31 01:15:47.844758+00 | t | active | 439006655 | 439000627 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24699 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52632 | 2018-05-31 00:50:03.41421+00 | 2018-05-31 01:15:50.071638+00 | 2018-05-31 01:15:50.541502+00 | 2018-05-31 01:15:50.541503+00 | t | active | 439006658 | 439000682 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24700 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52636 | 2018-05-31 00:50:03.434778+00 | 2018-05-31 01:16:38.025035+00 | 2018-05-31 01:16:38.490645+00 | 2018-05-31 01:16:38.490645+00 | t | active | 439007408 | 439002544 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24703 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52658 | 2018-05-31 00:50:03.476814+00 | 2018-05-31 01:13:51.305924+00 | 2018-05-31 01:13:51.762926+00 | 2018-05-31 01:13:51.762927+00 | t | active | 439006147 | 438991284 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24702 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52654 | 2018-05-31 00:50:03.476412+00 | 2018-05-31 01:11:12.982254+00 | 2018-05-31 01:11:13.506418+00 | 2018-05-31 01:11:13.506418+00 | t | active | 439004706 | 438980370 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24704 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52698 | 2018-05-31 00
:50:03.556858+00 | 2018-05-31 01:15:39.060058+00 | 2018-05-31 01:15:39.497926+00 | 2018-05-31 01:15:39.497926+00 | t | active | 439006634 | 439000535 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24705 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52702 | 2018-05-31 00:50:03.56562+00 | 2018-05-31 01:15:16.266355+00 | 2018-05-31 01:15:16.726527+00 | 2018-05-31 01:15:16.726528+00 | t | active | 439006585 | 439000374 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24706 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52720 | 2018-05-31 00:50:03.623633+00 | 2018-05-31 01:16:54.812318+00 | 2018-05-31 01:16:55.286572+00 | 2018-05-31 01:16:55.286573+00 | t | active | 439007486 | 439002773 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24707 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52760 | 2018-05-31 00:50:03.710674+00 | 2018-05-31 01:12:06.82123+00 | 2018-05-31 01:12:07.274585+00 | 2018-05-31 01:12:07.274586+00 | t | active | 439005264 | 438983414 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24708 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52774 | 2018-05-31 00:50:03.730793+00 | 2018-05-31 01:11:34.741754+00 | 2018-05-31 01:11:35.192976+00 | 2018-05-31 01:11:35.192977+00 | t | active | 439004756 | 438980639 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24716 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2170 | 2018-05-31 00:50:35.895793+00 | 2018-05-31 01:16:10.045597+00 | 2018-05-31 01:16:10.521668+00 | 2018-05-31 01:16:10.521669+00 | t | active | 439007355 | 439001292 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24717 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2172 | 2018-05-31 00:50:35.896245+00 | 2018-05-31 01:13:42.765309+00 | 2018-05-31 01:13:43.23401+00 | 2018-05-31 01:13:43.234011+00 | t | active | 439006135 | 438991207 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24720 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2198 | 2018-05-31 00:50:35.949632+00 | 2018-05-31 01:17:26.095977+00 | 2018-05-31 01:17:26.560013+00 | 2018-05-31 01:17:26.560013+00 | t | active | 439007913 | 439003773 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24722 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2224 | 2018-05-31 00:50:35.99574+00 | 2018-05-31 01:17:05.914558+00 | 2018-05-31 01:17:06.364112+00 | 2018-05-31 01:17:06.364112+00 | t | active | 439007644 | 439002877 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24723 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3120 | 2018-05-31 00:50:38.358004+00 | 2018-05-31 01:17:31.822477+00 | 2018-05-31 01:17:32.28692+00 | 2018-05-31 01:17:32.28692+00 | t | active | 439007920 | 439003781 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24724 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3122 | 2018-05-31 00:50:38.36036+00 | 2018-05-31 01:17:40.570731+00 | 2018-05-31 01:17:41.040108+00 | 2018-05-31 01:17:41.040109+00 | t | active | 439007939 | 439004000 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24726 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3358 | 2018-05-31 00:50:38.722412+00 | 2018-05-31 01:13:19.912358+00 | 2018-05-31 01:13:20.381085+00 | 2018-05-31 01:13:20.381085+00 | t | active | 439005992 | 438990225 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24729 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3364 | 2018-05-31 00:50:38.72951+00 | 2018-05-31 01:16:46.150281+00 | 2018-05-31 01:16:46.629381+00 | 2018-05-31 01:16:46.629381+00 | t | active | 439007417 | 439002551 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24728 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3362 | 2018-05-31 00:50:38.725571+00 | 2018-05-31 01:16:43.460621+00 | 2018-05-31 01:16:43.933521+00 | 2018-05-31 01:16:43.933522+00 | t | active | 439007414 | 439002548 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24731 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3404 | 2018-05-31 00
:50:38.792867+00 | 2018-05-31 01:11:03.793328+00 | 2018-05-31 01:11:04.263255+00 | 2018-05-31 01:11:04.263256+00 | t | active | 439004460 | 438980180 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24732 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 3832 | 2018-05-31 00:50:39.417184+00 | 2018-05-31 01:12:28.199226+00 | 2018-05-31 01:12:28.674556+00 | 2018-05-31 01:12:28.674557+00 | t | active | 439005476 | 438983980 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24743 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 6334 | 2018-05-31 00:50:43.61859+00 | 2018-05-31 01:17:09.802833+00 | 2018-05-31 01:17:10.237929+00 | 2018-05-31 01:17:10.23793+00 | t | active | 439007871 | 439002960 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24745 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 6366 | 2018-05-31 00:50:43.673628+00 | 2018-05-31 01:12:08.325086+00 | 2018-05-31 01:12:08.794176+00 | 2018-05-31 01:12:08.794177+00 | t | active | 439005368 | 438983638 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24750 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 6492 | 2018-05-31 00:50:43.923774+00 | 2018-05-31 01:09:56.342879+00 | 2018-05-31 01:09:56.787509+00 | 2018-05-31 01:09:56.787509+00 | t | active | 439004331 | 438976868 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24753 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 6600 | 2018-05-31 00:50:44.08386+00 | 2018-05-31 01:11:54.894164+00 | 2018-05-31 01:11:55.351153+00 | 2018-05-31 01:11:55.351153+00 | t | active | 439004841 | 438983262 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24759 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 6766 | 2018-05-31 00:50:44.442231+00 | 2018-05-31 01:14:59.258139+00 | 2018-05-31 01:14:59.717146+00 | 2018-05-31 01:14:59.717147+00 | t | active | 439006562 | 438998547 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24763 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7322 | 2018-05-31 00:50:45.312716+00 | 2018-05-31 01:17:14.931658+00 | 2018-05-31 01:17:15.379932+00 | 2018-05-31 01:17:15.379933+00 | t | active | 439007895 | 439003083 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24768 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7848 | 2018-05-31 00:50:46.169905+00 | 2018-05-31 01:13:05.761397+00 | 2018-05-31 01:13:06.206607+00 | 2018-05-31 01:13:06.206608+00 | t | active | 439005977 | 438987505 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24771 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 8028 | 2018-05-31 00:50:46.712831+00 | 2018-05-31 01:15:24.89299+00 | 2018-05-31 01:15:25.362258+00 | 2018-05-31 01:15:25.362259+00 | t | active | 439006598 | 439000413 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24801 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 20320 | 2018-05-31 00:51:15.994569+00 | 2018-05-31 01:10:22.141069+00 | 2018-05-31 01:10:22.588855+00 | 2018-05-31 01:10:22.588855+00 | t | active | 439004374 | 438979531 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24783 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 9244 | 2018-05-31 00:50:50.963901+00 | 2018-05-31 01:14:33.191062+00 | 2018-05-31 01:14:33.638441+00 | 2018-05-31 01:14:33.638456+00 | t | active | 439006275 | 438996997 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24787 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 10606 | 2018-05-31 00:50:54.555733+00 | 2018-05-31 01:15:27.584409+00 | 2018-05-31 01:15:28.034936+00 | 2018-05-31 01:15:28.034937+00 | t | active | 439006601 | 439000416 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24792 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 14162 | 2018-05-31 00:51:02.082733+00 | 2018-05-31 01:14:47.97586+00 | 2018-05-31 01:14:48.422684+00 | 2018-05-31 01:14:48.422685+00 | t | active | 439006543 | 438997392 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24799 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 18112 | 2018-05-31 00
:51:10.81511+00 | 2018-05-31 01:14:58.98876+00 | 2018-05-31 01:14:59.451169+00 | 2018-05-31 01:14:59.45117+00 | t | active | 439006560 | 438998547 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24798 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 18110 | 2018-05-31 00:51:10.81865+00 | 2018-05-31 01:16:01.542193+00 | 2018-05-31 01:16:01.967721+00 | 2018-05-31 01:16:01.967722+00 | t | active | 439007342 | 439000831 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24806 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 24284 | 2018-05-31 00:51:21.362725+00 | 2018-05-31 01:11:32.009778+00 | 2018-05-31 01:11:32.516554+00 | 2018-05-31 01:11:32.516555+00 | t | active | 439004752 | 438980587 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24810 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 27540 | 2018-05-31 00:51:25.779702+00 | 2018-05-31 01:17:37.759973+00 | 2018-05-31 01:17:38.220362+00 | 2018-05-31 01:17:38.220362+00 | t | active | 439007935 | 439003810 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24811 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 28566 | 2018-05-31 00:51:27.754166+00 | 2018-05-31 01:13:36.801662+00 | 2018-05-31 01:13:37.255387+00 | 2018-05-31 01:13:37.255388+00 | t | active | 439006127 | 438991155 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24813 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30386 | 2018-05-31 00:51:31.935543+00 | 2018-05-31 01:14:27.431723+00 | 2018-05-31 01:14:27.904266+00 | 2018-05-31 01:14:27.904267+00 | t | active | 439006261 | 438994479 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24816 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 32966 | 2018-05-31 00:51:38.06971+00 | 2018-05-31 01:15:07.549053+00 | 2018-05-31 01:15:07.997627+00 | 2018-05-31 01:15:07.997627+00 | t | active | 439006571 | 438998915 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24825 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 40802 | 2018-05-31 00:51:49.68438+00 | 2018-05-31 01:11:26.143804+00 | 2018-05-31 01:11:26.605007+00 | 2018-05-31 01:11:26.605008+00 | t | active | 439004726 | 438980463 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24828 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 47148 | 2018-05-31 00:52:01.34518+00 | 2018-05-31 01:16:01.714464+00 | 2018-05-31 01:16:02.178098+00 | 2018-05-31 01:16:02.178098+00 | t | active | 439007343 | 439000831 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24833 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 53336 | 2018-05-31 00:52:14.108793+00 | 2018-05-31 01:12:47.351075+00 | 2018-05-31 01:12:47.82186+00 | 2018-05-31 01:12:47.821861+00 | t | active | 439005927 | 438985264 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24834 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 53464 | 2018-05-31 00:52:14.416955+00 | 2018-05-31 01:14:18.55407+00 | 2018-05-31 01:14:19.017684+00 | 2018-05-31 01:14:19.017685+00 | t | active | 439006247 | 438994313 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24835 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 55602 | 2018-05-31 00:52:19.011905+00 | 2018-05-31 01:15:33.161734+00 | 2018-05-31 01:15:33.614617+00 | 2018-05-31 01:15:33.614617+00 | t | active | 439006607 | 439000437 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24839 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 60158 | 2018-05-31 00:52:27.43448+00 | 2018-05-31 01:14:36.14304+00 | 2018-05-31 01:14:36.595487+00 | 2018-05-31 01:14:36.595488+00 | t | active | 439006281 | 438997092 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24845 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2382 | 2018-05-31 00:52:38.920287+00 | 2018-05-31 01:14:18.821339+00 | 2018-05-31 01:14:19.284104+00 | 2018-05-31 01:14:19.284105+00 | t | active | 439006250 | 438994364 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24852 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2468 | 2018-05-31 00
:52:39.091814+00 | 2018-05-31 01:10:10.769804+00 | 2018-05-31 01:10:11.223708+00 | 2018-05-31 01:10:11.223709+00 | t | active | 439004358 | 438979273 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24868 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30408 | 2018-05-31 00:53:30.675444+00 | 2018-05-31 01:16:23.863678+00 | 2018-05-31 01:16:24.316163+00 | 2018-05-31 01:16:24.316164+00 | t | active | 439007374 | 439002257 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24864 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 15994 | 2018-05-31 00:53:05.240177+00 | 2018-05-31 01:14:09.27114+00 | 2018-05-31 01:14:09.738482+00 | 2018-05-31 01:14:09.738483+00 | t | active | 439006215 | 438993531 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24870 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30430 | 2018-05-31 00:53:30.697615+00 | 2018-05-31 01:09:41.772955+00 | 2018-05-31 01:09:42.231239+00 | 2018-05-31 01:09:42.23124+00 | t | active | 439004300 | 438975367 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24871 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30432 | 2018-05-31 00:53:30.704519+00 | 2018-05-31 01:10:55.753031+00 | 2018-05-31 01:10:56.216275+00 | 2018-05-31 01:10:56.216275+00 | t | active | 439004451 | 438980160 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24872 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30434 | 2018-05-31 00:53:30.705014+00 | 2018-05-31 01:16:12.745389+00 | 2018-05-31 01:16:13.191615+00 | 2018-05-31 01:16:13.191616+00 | t | active | 439007358 | 439001305 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24875 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30458 | 2018-05-31 00:53:30.737574+00 | 2018-05-31 01:11:51.709217+00 | 2018-05-31 01:11:52.168131+00 | 2018-05-31 01:11:52.168131+00 | t | active | 439004808 | 438983243 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24877 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30478 | 2018-05-31 00:53:30.777717+00 | 2018-05-31 01:09:30.605602+00 | 2018-05-31 01:09:31.054833+00 | 2018-05-31 01:09:31.054834+00 | t | active | 439004265 | 438974710 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24878 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30496 | 2018-05-31 00:53:30.828131+00 | 2018-05-31 01:11:32.013808+00 | 2018-05-31 01:11:32.462166+00 | 2018-05-31 01:11:32.462167+00 | t | active | 439004751 | 438980587 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24880 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30522 | 2018-05-31 00:53:30.883417+00 | 2018-05-31 01:10:13.638857+00 | 2018-05-31 01:10:14.119517+00 | 2018-05-31 01:10:14.119517+00 | t | active | 439004362 | 438979348 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24889 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 33250 | 2018-05-31 00:53:34.814292+00 | 2018-05-31 01:17:00.317816+00 | 2018-05-31 01:17:00.799471+00 | 2018-05-31 01:17:00.799472+00 | t | active | 439007497 | 439002815 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24888 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 33248 | 2018-05-31 00:53:34.813912+00 | 2018-05-31 01:16:31.921982+00 | 2018-05-31 01:16:32.39+00 | 2018-05-31 01:16:32.390001+00 | t | active | 439007396 | 439002285 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24893 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 35342 | 2018-05-31 00:53:40.329007+00 | 2018-05-31 01:17:04.860916+00 | 2018-05-31 01:17:05.331947+00 | 2018-05-31 01:17:05.331947+00 | t | active | 439007588 | 439002818 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24895 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 35352 | 2018-05-31 00:53:40.340711+00 | 2018-05-31 01:16:48.836582+00 | 2018-05-31 01:16:49.29966+00 | 2018-05-31 01:16:49.29966+00 | t | active | 439007420 | 439002554 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24897 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 35388 | 2018-05-31 00
:53:40.42175+00 | 2018-05-31 01:11:06.522691+00 | 2018-05-31 01:11:07.000439+00 | 2018-05-31 01:11:07.00044+00 | t | active | 439004463 | 438980211 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24902 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 39776 | 2018-05-31 00:53:46.807581+00 | 2018-05-31 01:17:14.704313+00 | 2018-05-31 01:17:15.185492+00 | 2018-05-31 01:17:15.185493+00 | t | active | 439007893 | 439003083 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24921 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 9892 | 2018-05-31 00:54:35.842792+00 | 2018-05-31 01:12:06.052698+00 | 2018-05-31 01:12:06.517976+00 | 2018-05-31 01:12:06.517977+00 | t | active | 439005189 | 438983414 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24920 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 9882 | 2018-05-31 00:54:35.839359+00 | 2018-05-31 01:12:47.665342+00 | 2018-05-31 01:12:48.139961+00 | 2018-05-31 01:12:48.139961+00 | t | active | 439005928 | 438985264 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24923 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 10790 | 2018-05-31 00:54:37.081963+00 | 2018-05-31 01:11:57.296347+00 | 2018-05-31 01:11:57.745705+00 | 2018-05-31 01:11:57.745706+00 | t | active | 439004854 | 438983279 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24947 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 49402 | 2018-05-31 00:55:30.868734+00 | 2018-05-31 01:15:30.308111+00 | 2018-05-31 01:15:30.77007+00 | 2018-05-31 01:15:30.770071+00 | t | active | 439006604 | 439000424 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24935 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 25862 | 2018-05-31 00:54:54.347118+00 | 2018-05-31 01:12:16.662117+00 | 2018-05-31 01:12:17.117229+00 | 2018-05-31 01:12:17.11723+00 | t | active | 439005461 | 438983866 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24955 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 58486 | 2018-05-31 00:55:46.385045+00 | 2018-05-31 01:09:59.227333+00 | 2018-05-31 01:09:59.685411+00 | 2018-05-31 01:09:59.685411+00 | t | active | 439004341 | 438978912 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24957 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 60654 | 2018-05-31 00:55:50.27037+00 | 2018-05-31 01:12:57.146592+00 | 2018-05-31 01:12:57.599951+00 | 2018-05-31 01:12:57.599951+00 | t | active | 439005965 | 438986648 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24972 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 24678 | 2018-05-31 00:56:40.155847+00 | 2018-05-31 01:12:10.036529+00 | 2018-05-31 01:12:10.503331+00 | 2018-05-31 01:12:10.503331+00 | t | active | 439005450 | 438983638 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25015 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 40766 | 2018-05-31 00:58:45.397651+00 | 2018-05-31 01:14:47.942668+00 | 2018-05-31 01:14:48.392929+00 | 2018-05-31 01:14:48.39293+00 | t | active | 439006541 | 438997392 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24977 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 30798 | 2018-05-31 00:56:52.711727+00 | 2018-05-31 01:10:07.882653+00 | 2018-05-31 01:10:08.36248+00 | 2018-05-31 01:10:08.362481+00 | t | active | 439004353 | 438978974 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24982 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 38966 | 2018-05-31 00:57:04.778824+00 | 2018-05-31 01:17:48.802422+00 | 2018-05-31 01:17:49.26832+00 | 2018-05-31 01:17:49.268321+00 | t | active | 439007949 | 439004038 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24983 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 42316 | 2018-05-31 00:57:09.764598+00 | 2018-05-31 01:12:51.37494+00 | 2018-05-31 01:12:51.851835+00 | 2018-05-31 01:12:51.851835+00 | t | active | 439005952 | 438986382 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24985 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 42678 | 2018-05-31 00
:57:10.506558+00 | 2018-05-31 01:14:18.945226+00 | 2018-05-31 01:14:19.38782+00 | 2018-05-31 01:14:19.387821+00 | t | active | 439006251 | 438994364 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24989 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 47382 | 2018-05-31 00:57:18.926677+00 | 2018-05-31 01:16:15.502202+00 | 2018-05-31 01:16:15.986875+00 | 2018-05-31 01:16:15.986875+00 | t | active | 439007361 | 439001322 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 24997 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 9158 | 2018-05-31 00:57:58.234245+00 | 2018-05-31 01:14:47.972945+00 | 2018-05-31 01:14:48.419138+00 | 2018-05-31 01:14:48.419139+00 | t | active | 439006542 | 438997392 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25001 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 22890 | 2018-05-31 00:58:17.073853+00 | 2018-05-31 01:13:11.4031+00 | 2018-05-31 01:13:11.861284+00 | 2018-05-31 01:13:11.861285+00 | t | active | 439005983 | 438989674 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25003 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 25486 | 2018-05-31 00:58:22.160883+00 | 2018-05-31 01:13:25.452643+00 | 2018-05-31 01:13:25.907029+00 | 2018-05-31 01:13:25.90703+00 | t | active | 439005999 | 438990360 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25005 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 27586 | 2018-05-31 00:58:25.764161+00 | 2018-05-31 01:09:33.984365+00 | 2018-05-31 01:09:34.456192+00 | 2018-05-31 01:09:34.456193+00 | t | active | 439004283 | 438974713 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25043 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 42204 | 2018-05-31 01:00:34.761688+00 | 2018-05-31 01:09:47.685798+00 | 2018-05-31 01:09:48.150269+00 | 2018-05-31 01:09:48.15027+00 | t | active | 439004315 | 438976591 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25051 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 49244 | 2018-05-31 01:00:47.586397+00 | 2018-05-31 01:17:23.239143+00 | 2018-05-31 01:17:23.703253+00 | 2018-05-31 01:17:23.703253+00 | t | active | 439007907 | 439003770 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25059 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2018 | 2018-05-31 01:01:16.096457+00 | 2018-05-31 01:16:00.075067+00 | 2018-05-31 01:16:00.53686+00 | 2018-05-31 01:16:00.536861+00 | t | active | 439007262 | 439000808 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25053 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 51836 | 2018-05-31 01:00:51.513606+00 | 2018-05-31 01:16:04.284806+00 | 2018-05-31 01:16:04.749427+00 | 2018-05-31 01:16:04.749428+00 | t | active | 439007346 | 439000890 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25086 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 52392 | 2018-05-31 01:02:39.791183+00 | 2018-05-31 01:11:28.932098+00 | 2018-05-31 01:11:29.387439+00 | 2018-05-31 01:11:29.38744+00 | t | active | 439004729 | 438980506 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25088 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 57978 | 2018-05-31 01:02:47.468491+00 | 2018-05-31 01:10:05.098154+00 | 2018-05-31 01:10:05.559502+00 | 2018-05-31 01:10:05.559503+00 | t | active | 439004350 | 438978959 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25123 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 39344 | 2018-05-31 01:04:41.65112+00 | 2018-05-31 01:13:39.674108+00 | 2018-05-31 01:13:40.149579+00 | 2018-05-31 01:13:40.14958+00 | t | active | 439006132 | 438991192 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25094 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 4596 | 2018-05-31 01:03:05.38285+00 | 2018-05-31 01:10:44.264293+00 | 2018-05-31 01:10:44.781187+00 | 2018-05-31 01:10:44.781187+00 | t | active | 439004438 | 438980062 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25096 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 12222 | 2018-05-31 01
:03:16.242928+00 | 2018-05-31 01:11:40.187702+00 | 2018-05-31 01:11:40.679487+00 | 2018-05-31 01:11:40.679487+00 | t | active | 439004762 | 438980672 | SELECT "containers".* FRO
M "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25101 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 19964 | 2018-05-31 01:03:33.807902+00 | 2018-05-31 01:12:16.534979+00 | 2018-05-31 01:12:17.065475+00 | 2018-05-31 01:12:17.065475+00 | t | active | 439005460 | 438983866 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25114 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 32876 | 2018-05-31 01:04:24.452117+00 | 2018-05-31 01:13:08.633955+00 | 2018-05-31 01:13:09.130296+00 | 2018-05-31 01:13:09.130296+00 | t | active | 439005980 | 438989162 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25158 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 47446 | 2018-05-31 01:05:33.183663+00 | 2018-05-31 01:14:21.724038+00 | 2018-05-31 01:14:22.225709+00 | 2018-05-31 01:14:22.225709+00 | t | active | 439006255 | 438994368 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25160 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 48698 | 2018-05-31 01:05:38.621653+00 | 2018-05-31 01:14:30.219709+00 | 2018-05-31 01:14:30.678552+00 | 2018-05-31 01:14:30.678553+00 | t | active | 439006264 | 438996060 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25172 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62166 | 2018-05-31 01:06:40.591423+00 | 2018-05-31 01:15:36.056806+00 | 2018-05-31 01:15:36.54356+00 | 2018-05-31 01:15:36.543561+00 | t | active | 439006612 | 439000502 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25174 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62230 | 2018-05-31 01:06:46.162275+00 | 2018-05-31 01:15:39.754375+00 | 2018-05-31 01:15:40.205664+00 | 2018-05-31 01:15:40.205665+00 | t | active | 439006639 | 439000535 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25175 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 62278 | 2018-05-31 01:06:49.02388+00 | 2018-05-31 01:15:41.659602+00 | 2018-05-31 01:15:42.126314+00 | 2018-05-31 01:15:42.126315+00 | t | active | 439006645 | 439000594 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25193 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 63938 | 2018-05-31 01:07:28.611025+00 | 2018-05-31 01:16:18.333508+00 | 2018-05-31 01:16:18.806338+00 | 2018-05-31 01:16:18.806339+00 | t | active | 439007367 | 439002098 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25194 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64116 | 2018-05-31 01:07:31.369684+00 | 2018-05-31 01:16:31.063264+00 | 2018-05-31 01:16:31.540546+00 | 2018-05-31 01:16:31.540547+00 | t | active | 439007394 | 439002263 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25196 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 64162 | 2018-05-31 01:07:34.350927+00 | 2018-05-31 01:16:26.520497+00 | 2018-05-31 01:16:26.98949+00 | 2018-05-31 01:16:26.98949+00 | t | active | 439007377 | 439002258 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25204 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 2374 | 2018-05-31 01:07:56.960432+00 | 2018-05-31 01:16:40.805951+00 | 2018-05-31 01:16:41.324118+00 | 2018-05-31 01:16:41.324119+00 | t | active | 439007411 | 439002545 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25237 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 6222 | 2018-05-31 01:09:22.271044+00 | 2018-05-31 01:18:02.874335+00 | 2018-05-31 01:18:03.398126+00 | 2018-05-31 01:18:03.398126+00 | t | active | 439007967 | 439004264 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25246 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 6950 | 2018-05-31 01:09:50.66842+00 | 2018-05-31 01:09:53.513079+00 | 2018-05-31 01:09:54.057617+00 | 2018-05-31 01:09:54.057617+00 | t | active | 439004326 | 438976628 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25254 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7272 | 2018-05-31 01:10:16.457782+00 | 2018-05-31 01:10:19.221228+00 | 2018-05-31 01:10:19.717579+00 | 2018-05-31 01:10:19.71758+00 | t | active | 439004370 | 438979495 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25258 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7326 | 2018-05-31 01:10:22.166461+00 | 2018-05-31 01:10:24.928751+00 | 2018-05-31 01:10:25.440367+00 | 2018-05-31 01:10:25.440368+00 | t | active | 439004379 | 438979710 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25260 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7350 | 2018-05-31 01:10:25.064066+00 | 2018-05-31 01:10:27.77752+00 | 2018-05-31 01:10:28.298566+00 | 2018-05-31 01:10:28.298567+00 | t | active | 439004383 | 438979718 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25265 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7456 | 2018-05-31 01:10:36.020409+00 | 2018-05-31 01:10:38.875034+00 | 2018-05-31 01:10:39.405111+00 | 2018-05-31 01:10:39.405112+00 | t | active | 439004432 | 438979945 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25266 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7490 | 2018-05-31 01:10:38.754879+00 | 2018-05-31 01:10:41.519756+00 | 2018-05-31 01:10:42.004406+00 | 2018-05-31 01:10:42.004406+00 | t | active | 439004435 | 438979980 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25268 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7518 | 2018-05-31 01:10:44.271458+00 | 2018-05-31 01:10:47.202904+00 | 2018-05-31 01:10:47.73973+00 | 2018-05-31 01:10:47.739731+00 | t | active | 439004441 | 438980089 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25270 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7552 | 2018-05-31 01:10:50.115074+00 | 2018-05-31 01:10:52.87549+00 | 2018-05-31 01:10:53.37698+00 | 2018-05-31 01:10:53.376981+00 | t | active | 439004447 | 438980152 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25272 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7606 | 2018-05-31 01:10:55.740711+00 | 2018-05-31 01:10:58.419907+00 | 2018-05-31 01:10:58.91862+00 | 2018-05-31 01:10:58.918621+00 | t | active | 439004454 | 438980172 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25273 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7626 | 2018-05-31 01:10:58.437426+00 | 2018-05-31 01:11:01.136394+00 | 2018-05-31 01:11:01.617621+00 | 2018-05-31 01:11:01.617622+00 | t | active | 439004457 | 438980179 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25284 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 7774 | 2018-05-31 01:11:09.322507+00 | 2018-05-31 01:11:12.661212+00 | 2018-05-31 01:11:13.168319+00 | 2018-05-31 01:11:13.16832+00 | t | active | 439004698 | 438980370 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25285 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 8084 | 2018-05-31 01:11:12.345083+00 | 2018-05-31 01:11:15.065234+00 | 2018-05-31 01:11:15.583254+00 | 2018-05-31 01:11:15.583255+00 | t | active | 439004713 | 438980416 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25288 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 8328 | 2018-05-31 01:11:20.664342+00 | 2018-05-31 01:11:23.417071+00 | 2018-05-31 01:11:23.916301+00 | 2018-05-31 01:11:23.916302+00 | t | active | 439004723 | 438980443 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
16385 | arvados_api_production | 25296 | 16384 | arvados | Passenger AppPreloader: /var/w...vados-api/current (forking...) | 10.101.0.30 | | 8556 | 2018-05-31 01:11:40.207355+00 | 2018-05-31 01:11:42.986227+00 | 2018-05-31 01:11:43.499448+00 | 2018-05-31 01:11:43.499449+00 | t | active | 439004777 | 438980935 | SELECT "containers".* FROM "containers" WHERE (uuid in ('eglyx-dz642-zissjywisjf1s66')) FOR UPDATE
(249 rows)
This causes requests to pile up in the passenger queue, eventually resulting in every single request failing due to timeout.
root@arvados-master-eglyx:/var/www/arvados-api/current/log# passenger-status | head -n14 Version : 5.3.1 Date : 2018-05-31 01:22:06 +0000 Instance: rtWsKXuN (nginx/1.14.0 Phusion_Passenger/5.3.1) ----------- General information ----------- Max pool size : 250 App groups : 1 Processes : 250 Requests in top-level queue : 0 ----------- Application groups ----------- /var/www/arvados-api/current (production): App root: /var/www/arvados-api/current Requests in queue: 2244
Updated by Joshua Randall almost 8 years ago
This is with arvados-api-server 1.1.4.20180509181910-8:
# dpkg -l |grep arvados-api ii arvados-api-server 1.1.4.20180509181910-8 amd64 Arvados API server - Arvados is a free and open source platform for big data science.
Updated by Joshua Randall almost 8 years ago
It seems to be a relatively small set of containers that tend to be the ones that end up locking up a large number of postgres connections. I've investigated a handful of them and they appear to be ones that are shared by a large number of workflows (i.e. common steps that are run at the beginning of the workflow to do things like prepare references and interval lists). So, there are many container_requests being submitted simultaneously that can all be fulfilled by the same container.
Updated by Joshua Randall almost 8 years ago
I'm only managing to keep our system running right now by occasionally killing postgres backends that are waiting on locks. I am running the following on the database server to kill ALL waiting backends if there are more than 50 of them (more than 20% of the 250 allowed connections):
# while (true); do kill_list=$(ps auxwww |grep 'arvados_api_prod' | grep waiting | awk '{print $2}'); if [[ $(echo "${kill_list}" | wc -l) -gt 50 ]]; then echo "too many waiting"; kill ${kill_list}; else echo "ok"; fi; sleep 60; done
Updated by Joshua Randall almost 8 years ago
we've now improved the killer script so it isn't so harsh:
$ while ( true ); do echo "select count(pid),string_agg(cast(pid as text), ' ') from pg_stat_activity where state = 'active' and waiting = 't' group by query;" | psql -t | awk 'BEGIN {FS="|"} $1>225 { print $2}' | while read killpids; do kill ${killpids} && echo "killed ${killpids}"; done; echo
"waiting 1m"; sleep 60; done
This basically says if more than 225 (90%) of the database connections are blocked waiting on a lock for exactly the same query, kill them all. If I don't do this, the API server becomes unresponsive and falls impossibly far behind the incoming requests.
The problem is not the locks per se, it is that they can block for a long time and while they are doing that consume all of the connections in the database, effectively acting as a DoS attack on other requests (which have nothing to do with the lock).
Updated by Peter Amstutz
Actions