Arvados

Background: These features will enable us to collect data about the impact of sharding on keep-balance time and memory requirements. Our expectation is that sharding will ultimately enable keep-balance to scale by a few more orders of magnitude, and decrease its resource consumption for all cluster sizes.

• If a hexadecimal prefix is specified on the command line (e.g., "f" or "0f"), keep-balance only balances blocks with that prefix.
  • This only makes sense as a data-gathering experiment using the -once flag.
  • Statistics will be misleading, because they will still have labels like "total" but only report a small percentage of the blocks.
  • At this stage we will not support an option to automatically run all shards sequentially or concurrently.
• Improve progress reporting while reading indexes from keepstore servers: e.g., every 5 minutes log "{volume}: received {N} blocks so far"
• Periodically report keep-balance's own memory usage