Project

General

Profile

Actions
Copy link

Idea #9494

closed

[SDKs] Python KeepClient copes better under memory pressure

Added by Brett Smith over 9 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
Story points:
-

Description

We received a user report that arv-put crashed with a MemoryError when uploading a 3GiB file from qr1hi's shell. This was apparently because of other work happening on the node at the time. Later attempts to upload the file succeeded.

Improve memory utilization in KeepClient. Possible relevant improvements include:

  • Use the Python buffer protocol to pass around data for upload
  • Hard limit the number of simultaneous running upload threads if they start failing their memory allocations
  • Generally audit and reduce the number of copies of data floating around - Other solutions suggested by profiling
Actions
Copy link
 #1

Updated by Brett Smith over 9 years ago

  • Subject changed from [SDKs] arv-put copes better under memory pressure to [SDKs] Python KeepClient copes better under memory pressure
  • Description updated (diff)
Actions
Copy link
 #2

Updated by Peter Amstutz about 6 years ago

  • Status changed from New to Closed
Actions
Copy link

Also available in: Atom PDF