Arvados

1. Keepstore PutBlock() calls log.Printf, this line of code is untouched from 2014 (!) it is being logged in JSON format but lacks useful context like the request id.

Updated "MD5 checksum %s did not match request" logs (and many other logs in keepstore) to use ctxlog so they include request id, loglevel, etc.

Fixed an unreported error: PutBlock tries volmgr.NextWritable(), and if that fails, it tries all writable volumes in sequence. The error from that first failure wasn't being logged.

1. The error that is sent to the client is not logged at all.

Already fixed in #15713.

1. The log doesn't say anything about where the block is being fetched from -- which volume, bucket, remote cluster, anything

If we get bad data from a remote service, we get two log entries:

msg="%s: MD5 checksum %s did not match request"

respStatusCode=502 respBody="checksum mismatch in remote response" (respBody added in #15713)

1. the error that reaches the user needs to make it clear that the problem was in fetching a remote block; requires some combination of improving server and client error messages

The server is sending "checksum mismatch in remote response".

15521-keepstore-logging @ 62d28600cbfc31f8e72c61e4519ff198cb66a02a -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1616/

This LGTM, thanks.