Arvados

Current implementation of Keep @ a4378cd handles both GET and PUT requests by reading an entire blob (up to 64MB) into memory, performing checksums and any other policy checks, and then delivering the block appropriately. With many concurrent clients, this can bring Keep memory usage up to 500MB and more.

A more memory-efficient solution, and lower latency from the client's point of view, would be to stream data between the client and disk as much as possible with smaller buffers. The keepclient package has some useful tools for this.

• When --serialize-io is enabled, Keep must continue to read an entire block and buffer it, or else a single slow client may block other clients while they wait for disk access.
• GET cannot issue a 402 Corruption error without hashing the block first. Possible solutions include:
  • Read the block twice through a small buffer: once to calculate the checksum, and again to stream it to the client.
  • Deliver any corruption status in the second part of a multipart/mixed message.
  • Abandon any attempt to report corruption to the client and simply log it to the data manager, on the grounds that the client is already responsible for checking for corrupt blocks anyway.
• PUT checking for collisions, which currently relies on GetBlock, will need to be rethought to handle streaming data.