Bug #19192: WebDAVCache not performing as expected for S3 requests - Arvados

Actions

Bug #19192

closed

WebDAVCache not performing as expected for S3 requests

Added by Tom Clegg over 2 years ago. Updated over 2 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Category:

Keep

Target version:

Start date:

06/21/2022

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Story points:

-

Release:

Release relationship:

Auto

Files

Download all files

keep-web (11 MB) keep-web	dev build at commit 042f0d37dacebddef9d2d7232fa2e5b9922a9451	Tom Clegg, 06/13/2022 06:15 PM
keep-web (11 MB) keep-web	dev build at commit 65c1517193b5039769f288e84a11d1ebfddbf031	Tom Clegg, 06/20/2022 04:44 PM
keep-web (11 MB) keep-web	dev build at commit b5db4b70878a3907db3691b71979e6e65511a12c	Tom Clegg, 06/21/2022 03:22 AM

Subtasks 1 (0 open — 1 closed)

Related issues 2 (0 open — 2 closed)

Actions

#1

Updated by Tom Clegg over 2 years ago

Status changed from New to In Progress

Actions

#2

Updated by Tom Clegg over 2 years ago

File keep-web keep-web added

Actions

#3

Updated by Tom Clegg over 2 years ago

Debug logging indicates (*cache)pruneSessions() is not running, which causes expired sessions to get stuck in the cache forever. I'm guessing there's a situation where fs.MemorySize() deadlocks, and once that happens, pruneSessions can't continue/resume, and we effectively have no cache.

SIGABRT would produce a stack trace to confirm this and show where it's getting stuck.

Actions

#4

Updated by Tom Clegg over 2 years ago

File keep-web keep-web added

Here's a debug version that will dump a stack trace and exit if cache pruning takes longer than 30s.

Actions

#5

Updated by Tom Clegg over 2 years ago

File keep-web added

19192-fix-deadlock @ dab0c5596e39dc455d88bba797717e829fe5caf5 -- developer-run-tests: #3177

Fixes a lock-ordering bug in lookupnode causing deadlock on concurrent open() and readdir() calls in the same lookupnode (project) directory.

Actions

#6

Updated by Tom Clegg over 2 years ago

File deleted (~~keep-web~~)

Actions

#7

Updated by Tom Clegg over 2 years ago

File keep-web keep-web added

Here's a dev build with both the bugfix from note-5 and the watchdog / stack dump from note-4.

Actions

#8

Updated by Lucas Di Pentima over 2 years ago

19192-fix-deadlock LGTM, thanks!

Actions

#9

Updated by Peter Amstutz over 2 years ago

Target version changed from 2022-06-22 Sprint to 2022-07-06

Actions

#10

Updated by Tom Clegg over 2 years ago

Related to Story #19205: In Go services, monitor request times and record when they continue processing after client disconnects, or exceed a maximum request time added

Actions

#11

Updated by Peter Amstutz over 2 years ago

Target version changed from 2022-07-06 to 2022-07-20

Actions

#12

Updated by Peter Amstutz over 2 years ago

Target version changed from 2022-07-20 to 2022-07-06

Actions

#13

Updated by Peter Amstutz over 2 years ago

Category set to Keep
Status changed from In Progress to Resolved

Actions

#14

Updated by Peter Amstutz over 2 years ago

Release set to 52

Actions

#15

Updated by Tom Clegg over 2 years ago

Related to Bug #19368: [keep-web] [S3] slow requests caused by logUploadOrDownload added

Actions

Also available in: Atom PDF