Bug #13407: [keep-balance] Take keepstore-reported volume replication level into account - Arvados

Custom queries

All assigned issues
All issues assigned for grooming
My issues for grooming
My issues for grooming (no story pts)
Prioritized open issues

Actions

Copy link

Bug #13407

closed

[keep-balance] Take keepstore-reported volume replication level into account

Added by Tom Clegg over 6 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Tom Clegg

Category:

Target version:

2018-06-20 Sprint

Start date:

06/11/2018

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Story points:

1.0

Release:

Arvados 1.2

Release relationship:

Auto

Description

Currently keepstore reports the replication level configured for each volume, but keep-balance ignores this and treats every volume as replication=1. With the default API configuration (default_collection_replication=2) this causes keep-balance to store replicas on 2 volumes, even if all volumes advertise replication≥2 themselves.

Logs/reports should continue to count a stored copy as "1 replica" even if the underlying volume reports replication>1.

Subtasks 1 (0 open — 1 closed)

Task #13521: Review 13407-volume-replication

Resolved

Ward Vandewege

06/11/2018

Actions

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Tom Clegg over 6 years ago

Description updated (diff)

Implementation notes

In (*Balancer)balanceBlock(), track replication_achieved_so_far, instead of len(mounts_used_so_far)
Update any existing tests that depend on keep-balance ignoring volume replication>1
Add tests
- desire 3, all mounts offer 2 → two replicas
- desire 2, all mounts offer 3 → one replica
- desire 3, best position offers 1, next-best position offers 3 → either two replicas, or skip the best position (both would be acceptable, but it's not necessary to optimize for this kind of setup now -- for now use two replicas if that's substantially easier)

Actions

Copy link

Updated by Tom Clegg over 6 years ago

Story points set to 1.0

Actions

Copy link

Updated by Tom Morris over 6 years ago

Target version changed from To Be Groomed to Arvados Future Sprints

Actions

Copy link

Updated by Tom Morris over 6 years ago

Target version changed from Arvados Future Sprints to 2018-06-06 Sprint

Actions

Copy link

Updated by Tom Morris over 6 years ago

Assigned To set to Tom Clegg

Actions

Copy link

Updated by Tom Clegg over 6 years ago

Target version changed from 2018-06-06 Sprint to 2018-06-20 Sprint

Actions

Copy link

Updated by Tom Clegg over 6 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Tom Clegg over 6 years ago

13407-volume-replication @ 06eb98871e82074b935d9da201118119e5719f96

Actions

Copy link

#10

Updated by Ward Vandewege over 6 years ago

Running on 06eb98871e82074b935d9da201118119e5719f96 on branch 13407-volume-replication, in the 'every keepstore sees every volume rw' scenario:

Jun 11 20:24:41 dhhck.arvadosapi.com systemd[1]: Started Arvados Keep Balance.
Jun 11 20:24:41 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:41 keep-balance dev started
Jun 11 20:24:41 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:41 starting up: will scan every 4h0m0s and on SIGUSR1
Jun 11 20:24:41 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:41 WARNING: Will scan periodically, but no changes will be committed.
Jun 11 20:24:41 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:41 =======  Consider using -commit-pulls and -commit-trash flags.
Jun 11 20:24:41 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:41 Run: start
Jun 11 20:24:42 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:42 skipping dhhck-bi6l4-blb6dtx0sx07d7k with service type "proxy" 
Jun 11 20:24:42 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:42 GetCurrentState: start
Jun 11 20:24:42 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:42 mount zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3)
Jun 11 20:24:42 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:42 mount zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3)
Jun 11 20:24:42 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:42 mount zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3)
Jun 11 20:24:42 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:42 mount zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3)
Jun 11 20:24:43 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:43 collections: 0/137060
Jun 11 20:24:45 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:45 collections: 1000/137060
Jun 11 20:24:46 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:46 collections: 2000/137060
Jun 11 20:24:47 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:47 collections: 3000/137060
Jun 11 20:24:48 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:48 collections: 4000/137060
Jun 11 20:24:50 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:50 collections: 5000/137060
Jun 11 20:24:50 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:50 collections: 6000/137060
Jun 11 20:24:52 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:52 collections: 7000/137060
Jun 11 20:24:52 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:52 collections: 8000/137060
Jun 11 20:24:53 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:53 collections: 9000/137060
Jun 11 20:24:54 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:54 collections: 10000/137060
Jun 11 20:24:55 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:55 collections: 11000/137060
Jun 11 20:24:55 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:55 collections: 12000/137060
Jun 11 20:24:56 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:56 collections: 13000/137060
Jun 11 20:24:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:57 collections: 14000/137060
Jun 11 20:24:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:58 collections: 15000/137060
Jun 11 20:24:59 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:59 collections: 16000/137060
Jun 11 20:24:59 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:24:59 collections: 17000/137060
Jun 11 20:25:00 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:00 collections: 18000/137060
Jun 11 20:25:01 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:01 collections: 19000/137060
Jun 11 20:25:04 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:04 collections: 20000/137060
Jun 11 20:25:12 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:12 collections: 21000/137060
Jun 11 20:25:15 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:15 collections: 22000/137060
Jun 11 20:25:18 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:18 collections: 23000/137060
Jun 11 20:25:19 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:19 collections: 24000/137060
Jun 11 20:25:20 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:20 collections: 25000/137060
Jun 11 20:25:21 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:21 collections: 26000/137060
Jun 11 20:25:21 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:21 collections: 27000/137060
Jun 11 20:25:22 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:22 collections: 28000/137060
Jun 11 20:25:24 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:24 collections: 29000/137060
Jun 11 20:25:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:25 collections: 30000/137060
Jun 11 20:25:26 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:26 collections: 31000/137060
Jun 11 20:25:27 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:27 collections: 32000/137060
Jun 11 20:25:28 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:28 collections: 33000/137060
Jun 11 20:25:29 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:29 collections: 34000/137060
Jun 11 20:25:30 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:30 collections: 35000/137060
Jun 11 20:25:31 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:31 collections: 36000/137060
Jun 11 20:25:32 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:32 collections: 37000/137060
Jun 11 20:25:33 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:33 collections: 38000/137060
Jun 11 20:25:34 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:34 collections: 39000/137060
Jun 11 20:25:35 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:35 collections: 40000/137060
Jun 11 20:25:36 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:36 collections: 41000/137060
Jun 11 20:25:37 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:37 collections: 42000/137060
Jun 11 20:25:38 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:38 collections: 43000/137060
Jun 11 20:25:39 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:39 collections: 44000/137060
Jun 11 20:25:40 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:40 collections: 45000/137060
Jun 11 20:25:41 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:41 collections: 46000/137060
Jun 11 20:25:42 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:42 collections: 47000/137060
Jun 11 20:25:43 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:43 collections: 48000/137060
Jun 11 20:25:44 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:44 collections: 49000/137060
Jun 11 20:25:44 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:44 collections: 50000/137060
Jun 11 20:25:46 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:46 collections: 51000/137060
Jun 11 20:25:47 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:47 collections: 52000/137060
Jun 11 20:25:48 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:48 collections: 53000/137060
Jun 11 20:25:48 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:48 collections: 54000/137060
Jun 11 20:25:49 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:49 collections: 55000/137060
Jun 11 20:25:50 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:50 collections: 56000/137060
Jun 11 20:25:51 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:51 collections: 57000/137060
Jun 11 20:25:52 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:52 collections: 58000/137060
Jun 11 20:25:54 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:54 collections: 59000/137060
Jun 11 20:25:54 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:54 collections: 60000/137060
Jun 11 20:25:55 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:55 collections: 61000/137060
Jun 11 20:25:56 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:56 collections: 62000/137060
Jun 11 20:25:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:57 collections: 63000/137060
Jun 11 20:25:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:58 collections: 64000/137060
Jun 11 20:25:59 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:25:59 collections: 65000/137060
Jun 11 20:26:00 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:00 collections: 66000/137060
Jun 11 20:26:01 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:01 collections: 67000/137060
Jun 11 20:26:02 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:02 collections: 68000/137060
Jun 11 20:26:03 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:03 collections: 69000/137060
Jun 11 20:26:04 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:04 collections: 70000/137060
Jun 11 20:26:05 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:05 collections: 71000/137060
Jun 11 20:26:06 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:06 collections: 72000/137060
Jun 11 20:26:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:08 collections: 73000/137060
Jun 11 20:26:09 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:09 collections: 74000/137060
Jun 11 20:26:09 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:09 collections: 75000/137060
Jun 11 20:26:10 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:10 collections: 76000/137060
Jun 11 20:26:12 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:12 collections: 77000/137060
Jun 11 20:26:13 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:13 collections: 78000/137060
Jun 11 20:26:13 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:13 collections: 79000/137060
Jun 11 20:26:14 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:14 collections: 80000/137060
Jun 11 20:26:15 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:15 collections: 81000/137060
Jun 11 20:26:16 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:16 collections: 82000/137060
Jun 11 20:26:17 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:17 collections: 83000/137060
Jun 11 20:26:18 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:18 collections: 84000/137060
Jun 11 20:26:19 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:19 collections: 85000/137060
Jun 11 20:26:20 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:20 collections: 86000/137060
Jun 11 20:26:21 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:21 collections: 87000/137060
Jun 11 20:26:22 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:22 collections: 88000/137060
Jun 11 20:26:22 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:22 collections: 89000/137060
Jun 11 20:26:23 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:23 collections: 90000/137060
Jun 11 20:26:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:25 collections: 91000/137060
Jun 11 20:26:26 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:26 collections: 92000/137060
Jun 11 20:26:27 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:27 collections: 93000/137060
Jun 11 20:26:27 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:27 collections: 94000/137060
Jun 11 20:26:28 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:28 collections: 95000/137060
Jun 11 20:26:29 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:29 collections: 96000/137060
Jun 11 20:26:30 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:30 collections: 97000/137060
Jun 11 20:26:30 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:30 collections: 98000/137060
Jun 11 20:26:31 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:31 collections: 99000/137060
Jun 11 20:26:32 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:32 collections: 100000/137060
Jun 11 20:26:33 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:33 collections: 101000/137060
Jun 11 20:26:33 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:33 collections: 102000/137060
Jun 11 20:26:34 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:34 collections: 103000/137060
Jun 11 20:26:35 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:35 collections: 104000/137060
Jun 11 20:26:36 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:36 collections: 105000/137060
Jun 11 20:26:37 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:37 collections: 106000/137060
Jun 11 20:26:37 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:37 collections: 107000/137060
Jun 11 20:26:38 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:38 collections: 108000/137060
Jun 11 20:26:39 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:39 collections: 109000/137060
Jun 11 20:26:40 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:40 collections: 110000/137060
Jun 11 20:26:40 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:40 collections: 111000/137060
Jun 11 20:26:41 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:41 collections: 112000/137060
Jun 11 20:26:42 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:42 collections: 113000/137060
Jun 11 20:26:43 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:43 collections: 114000/137060
Jun 11 20:26:43 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:43 collections: 115000/137060
Jun 11 20:26:44 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:44 collections: 116000/137060
Jun 11 20:26:45 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:45 collections: 117000/137060
Jun 11 20:26:46 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:46 collections: 118000/137060
Jun 11 20:26:46 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:46 collections: 119000/137060
Jun 11 20:26:47 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:47 collections: 120000/137060
Jun 11 20:26:48 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:48 collections: 121000/137060
Jun 11 20:26:49 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:49 collections: 122000/137060
Jun 11 20:26:49 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:49 collections: 123000/137060
Jun 11 20:26:50 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:50 collections: 124000/137060
Jun 11 20:26:51 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:51 collections: 125000/137060
Jun 11 20:26:52 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:52 collections: 126000/137060
Jun 11 20:26:52 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:52 collections: 127000/137060
Jun 11 20:26:53 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:53 collections: 128000/137060
Jun 11 20:26:54 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:54 collections: 129000/137060
Jun 11 20:26:55 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:55 collections: 130000/137060
Jun 11 20:26:55 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:55 collections: 131000/137060
Jun 11 20:26:56 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:56 collections: 132000/137060
Jun 11 20:26:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:57 collections: 133000/137060
Jun 11 20:26:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:58 collections: 134000/137060
Jun 11 20:26:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:26:58 collections: 135000/137060
Jun 11 20:27:00 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:27:00 collections: 136000/137060
Jun 11 20:27:03 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:27:03 collections: 137000/137060
Jun 11 20:27:03 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:27:03 collections: 137060/137060
Jun 11 20:27:03 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:27:03 collections: 137060/137060
Jun 11 20:31:07 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:07 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map
Jun 11 20:31:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:08 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 512782 replicas
Jun 11 20:31:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:08 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map
Jun 11 20:31:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:08 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 512782 replicas
Jun 11 20:31:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:08 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map
Jun 11 20:31:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:08 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 512782 replicas
Jun 11 20:31:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:08 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map
Jun 11 20:31:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:08 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 512782 replicas
Jun 11 20:31:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:31:08 mount zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): index done
Jun 11 20:33:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:57 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map
Jun 11 20:33:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:58 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 898276 replicas
Jun 11 20:33:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:58 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map
Jun 11 20:33:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:58 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 898276 replicas
Jun 11 20:33:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:58 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map
Jun 11 20:33:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:58 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 898276 replicas
Jun 11 20:33:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:58 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map
Jun 11 20:33:59 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:59 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 898276 replicas
Jun 11 20:33:59 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:33:59 mount zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): index done
Jun 11 20:34:07 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:07 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map
Jun 11 20:34:07 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:07 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 902720 replicas
Jun 11 20:34:07 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:07 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map
Jun 11 20:34:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:08 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 902720 replicas
Jun 11 20:34:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:08 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map
Jun 11 20:34:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:08 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 902720 replicas
Jun 11 20:34:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:08 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map
Jun 11 20:34:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:08 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 902720 replicas
Jun 11 20:34:08 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:34:08 mount zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): index done
Jun 11 20:35:56 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:56 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map
Jun 11 20:35:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:57 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas
Jun 11 20:35:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:57 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map
Jun 11 20:35:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:57 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas
Jun 11 20:35:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:57 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map
Jun 11 20:35:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:57 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas
Jun 11 20:35:57 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:57 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map
Jun 11 20:35:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:58 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas
Jun 11 20:35:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:58 mount zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): index done
Jun 11 20:35:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:58 GetCurrentState: took 11m16.001718919s
Jun 11 20:35:58 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:35:58 ComputeChangeSets: start
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 ComputeChangeSets: took 27.015471439s
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 ===
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 0 replicas (0 blocks, 0 bytes) lost (0=have<want)
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 0 replicas (0 blocks, 0 bytes) underreplicated (0<have<want)
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 3357390 replicas (3357390 blocks, 199210561559859 bytes) just right (have=want)
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 154 replicas (150 blocks, 7123912145 bytes) overreplicated (have>want>0)
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new)
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old)
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 ===
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 storage class "default": 6715080 replicas (3357540 blocks, 398423158486764 bytes) desired
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 storage class "default": 3357390 replicas (3357390 blocks, 199210561559859 bytes) short
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 storage class "default": 4 replicas (4 blocks, 31871 bytes) surplus
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 storage class "default": 0 replicas (0 blocks, 0 bytes) unachievable
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 ===
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 3357540 replicas (3357540 blocks, 199211579243382 bytes) total commitment (excluding unreferenced)
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 13430776 replicas (3357540 blocks, 796850387835104 bytes) total usage
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 ===
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:0, Trashes:0}
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:0, Trashes:0}
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:0, Trashes:0}
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:0, Trashes:0}
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 ===
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 Replication level distribution (counting N replicas on a single server as N):
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  0:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  1:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  2:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  3:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  4: 3357390 ###########################################################
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  5:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  6:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  7:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  8:     146 ###################
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25  9:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 10:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 11:       0
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 12:       4 ######
Jun 11 20:36:25 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:25 ===
Jun 11 20:36:26 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:26 Run: took 11m44.964962457s
Jun 11 20:36:26 dhhck.arvadosapi.com keep-balance[11188]: 2018/06/11 20:36:26 run succeeded

Actions

Copy link

#11

Updated by Tom Clegg over 6 years ago

13407-volume-replication @ 8fa57330875af11b7b61dfac6fc45080760afdf2

fixes stats

Actions

Copy link

#12

Updated by Ward Vandewege over 6 years ago

With 8fa57330875af11b7b61dfac6fc45080760afdf2 on branch 13407-volume-replication, in the 'every keepstore sees every volume rw' scenario:

Jun 11 21:05:46 dhhck.arvadosapi.com systemd[1]: Started Arvados Keep Balance.
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 keep-balance dev started
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 starting up: will scan every 4h0m0s and on SIGUSR1
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 WARNING: Will scan periodically, but no changes will be committed.
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 =======  Consider using -commit-pulls and -commit-trash flags.
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 Run: start
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 skipping dhhck-bi6l4-blb6dtx0sx07d7k with service type "proxy" 
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 GetCurrentState: start
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 mount zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3)
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 mount zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3)
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 mount zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3)
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 mount zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3)
Jun 11 21:05:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:46 collections: 0/137060
Jun 11 21:05:47 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:47 collections: 1000/137060
Jun 11 21:05:48 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:48 collections: 2000/137060
Jun 11 21:05:48 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:48 collections: 3000/137060
Jun 11 21:05:49 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:49 collections: 4000/137060
Jun 11 21:05:51 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:51 collections: 5000/137060
Jun 11 21:05:51 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:51 collections: 6000/137060
Jun 11 21:05:52 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:52 collections: 7000/137060
Jun 11 21:05:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:53 collections: 8000/137060
Jun 11 21:05:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:53 collections: 9000/137060
Jun 11 21:05:54 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:54 collections: 10000/137060
Jun 11 21:05:55 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:55 collections: 11000/137060
Jun 11 21:05:56 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:56 collections: 12000/137060
Jun 11 21:05:57 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:57 collections: 13000/137060
Jun 11 21:05:58 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:58 collections: 14000/137060
Jun 11 21:05:58 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:58 collections: 15000/137060
Jun 11 21:05:59 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:05:59 collections: 16000/137060
Jun 11 21:06:00 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:00 collections: 17000/137060
Jun 11 21:06:01 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:01 collections: 18000/137060
Jun 11 21:06:01 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:01 collections: 19000/137060
Jun 11 21:06:03 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:03 collections: 20000/137060
Jun 11 21:06:05 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:05 collections: 21000/137060
Jun 11 21:06:08 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:08 collections: 22000/137060
Jun 11 21:06:09 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:09 collections: 23000/137060
Jun 11 21:06:09 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:09 collections: 24000/137060
Jun 11 21:06:10 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:10 collections: 25000/137060
Jun 11 21:06:11 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:11 collections: 26000/137060
Jun 11 21:06:12 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:12 collections: 27000/137060
Jun 11 21:06:13 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:13 collections: 28000/137060
Jun 11 21:06:14 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:14 collections: 29000/137060
Jun 11 21:06:15 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:15 collections: 30000/137060
Jun 11 21:06:15 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:15 collections: 31000/137060
Jun 11 21:06:16 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:16 collections: 32000/137060
Jun 11 21:06:17 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:17 collections: 33000/137060
Jun 11 21:06:18 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:18 collections: 34000/137060
Jun 11 21:06:19 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:19 collections: 35000/137060
Jun 11 21:06:20 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:20 collections: 36000/137060
Jun 11 21:06:21 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:21 collections: 37000/137060
Jun 11 21:06:22 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:22 collections: 38000/137060
Jun 11 21:06:23 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:23 collections: 39000/137060
Jun 11 21:06:24 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:24 collections: 40000/137060
Jun 11 21:06:25 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:25 collections: 41000/137060
Jun 11 21:06:26 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:26 collections: 42000/137060
Jun 11 21:06:27 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:27 collections: 43000/137060
Jun 11 21:06:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:28 collections: 44000/137060
Jun 11 21:06:29 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:29 collections: 45000/137060
Jun 11 21:06:29 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:29 collections: 46000/137060
Jun 11 21:06:30 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:30 collections: 47000/137060
Jun 11 21:06:31 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:31 collections: 48000/137060
Jun 11 21:06:32 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:32 collections: 49000/137060
Jun 11 21:06:33 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:33 collections: 50000/137060
Jun 11 21:06:34 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:34 collections: 51000/137060
Jun 11 21:06:35 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:35 collections: 52000/137060
Jun 11 21:06:36 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:36 collections: 53000/137060
Jun 11 21:06:37 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:37 collections: 54000/137060
Jun 11 21:06:38 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:38 collections: 55000/137060
Jun 11 21:06:39 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:39 collections: 56000/137060
Jun 11 21:06:40 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:40 collections: 57000/137060
Jun 11 21:06:41 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:41 collections: 58000/137060
Jun 11 21:06:42 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:42 collections: 59000/137060
Jun 11 21:06:43 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:43 collections: 60000/137060
Jun 11 21:06:44 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:44 collections: 61000/137060
Jun 11 21:06:44 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:44 collections: 62000/137060
Jun 11 21:06:45 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:45 collections: 63000/137060
Jun 11 21:06:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:46 collections: 64000/137060
Jun 11 21:06:47 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:47 collections: 65000/137060
Jun 11 21:06:48 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:48 collections: 66000/137060
Jun 11 21:06:49 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:49 collections: 67000/137060
Jun 11 21:06:50 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:50 collections: 68000/137060
Jun 11 21:06:51 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:51 collections: 69000/137060
Jun 11 21:06:52 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:52 collections: 70000/137060
Jun 11 21:06:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:53 collections: 71000/137060
Jun 11 21:06:54 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:54 collections: 72000/137060
Jun 11 21:06:55 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:55 collections: 73000/137060
Jun 11 21:06:56 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:56 collections: 74000/137060
Jun 11 21:06:57 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:57 collections: 75000/137060
Jun 11 21:06:58 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:58 collections: 76000/137060
Jun 11 21:06:59 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:06:59 collections: 77000/137060
Jun 11 21:07:00 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:00 collections: 78000/137060
Jun 11 21:07:01 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:01 collections: 79000/137060
Jun 11 21:07:02 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:02 collections: 80000/137060
Jun 11 21:07:02 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:02 collections: 81000/137060
Jun 11 21:07:03 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:03 collections: 82000/137060
Jun 11 21:07:04 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:04 collections: 83000/137060
Jun 11 21:07:05 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:05 collections: 84000/137060
Jun 11 21:07:06 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:06 collections: 85000/137060
Jun 11 21:07:07 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:07 collections: 86000/137060
Jun 11 21:07:08 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:08 collections: 87000/137060
Jun 11 21:07:09 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:09 collections: 88000/137060
Jun 11 21:07:09 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:09 collections: 89000/137060
Jun 11 21:07:10 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:10 collections: 90000/137060
Jun 11 21:07:12 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:12 collections: 91000/137060
Jun 11 21:07:12 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:12 collections: 92000/137060
Jun 11 21:07:13 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:13 collections: 93000/137060
Jun 11 21:07:14 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:14 collections: 94000/137060
Jun 11 21:07:15 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:15 collections: 95000/137060
Jun 11 21:07:15 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:15 collections: 96000/137060
Jun 11 21:07:16 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:16 collections: 97000/137060
Jun 11 21:07:17 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:17 collections: 98000/137060
Jun 11 21:07:17 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:17 collections: 99000/137060
Jun 11 21:07:18 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:18 collections: 100000/137060
Jun 11 21:07:19 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:19 collections: 101000/137060
Jun 11 21:07:19 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:19 collections: 102000/137060
Jun 11 21:07:20 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:20 collections: 103000/137060
Jun 11 21:07:21 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:21 collections: 104000/137060
Jun 11 21:07:22 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:22 collections: 105000/137060
Jun 11 21:07:22 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:22 collections: 106000/137060
Jun 11 21:07:23 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:23 collections: 107000/137060
Jun 11 21:07:24 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:24 collections: 108000/137060
Jun 11 21:07:24 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:24 collections: 109000/137060
Jun 11 21:07:25 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:25 collections: 110000/137060
Jun 11 21:07:26 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:26 collections: 111000/137060
Jun 11 21:07:27 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:27 collections: 112000/137060
Jun 11 21:07:27 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:27 collections: 113000/137060
Jun 11 21:07:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:28 collections: 114000/137060
Jun 11 21:07:29 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:29 collections: 115000/137060
Jun 11 21:07:30 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:30 collections: 116000/137060
Jun 11 21:07:30 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:30 collections: 117000/137060
Jun 11 21:07:31 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:31 collections: 118000/137060
Jun 11 21:07:32 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:32 collections: 119000/137060
Jun 11 21:07:32 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:32 collections: 120000/137060
Jun 11 21:07:33 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:33 collections: 121000/137060
Jun 11 21:07:34 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:34 collections: 122000/137060
Jun 11 21:07:34 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:34 collections: 123000/137060
Jun 11 21:07:35 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:35 collections: 124000/137060
Jun 11 21:07:36 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:36 collections: 125000/137060
Jun 11 21:07:36 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:36 collections: 126000/137060
Jun 11 21:07:37 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:37 collections: 127000/137060
Jun 11 21:07:38 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:38 collections: 128000/137060
Jun 11 21:07:39 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:39 collections: 129000/137060
Jun 11 21:07:39 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:39 collections: 130000/137060
Jun 11 21:07:40 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:40 collections: 131000/137060
Jun 11 21:07:41 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:41 collections: 132000/137060
Jun 11 21:07:41 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:41 collections: 133000/137060
Jun 11 21:07:42 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:42 collections: 134000/137060
Jun 11 21:07:43 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:43 collections: 135000/137060
Jun 11 21:07:44 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:44 collections: 136000/137060
Jun 11 21:07:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:46 collections: 137000/137060
Jun 11 21:07:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:46 collections: 137060/137060
Jun 11 21:07:46 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:07:46 collections: 137060/137060
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 512782 replicas
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 512782 replicas
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 512782 replicas
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 512782 replicas
Jun 11 21:12:28 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:12:28 mount zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): index done
Jun 11 21:14:54 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:54 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map
Jun 11 21:14:55 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:55 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 898276 replicas
Jun 11 21:14:55 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:55 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map
Jun 11 21:14:55 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:55 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 898276 replicas
Jun 11 21:14:55 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:55 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map
Jun 11 21:14:56 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:56 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 898276 replicas
Jun 11 21:14:56 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:56 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map
Jun 11 21:14:56 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:56 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 898276 replicas
Jun 11 21:14:56 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:14:56 mount zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): index done
Jun 11 21:15:15 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:15 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map
Jun 11 21:15:16 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:16 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 902720 replicas
Jun 11 21:15:16 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:16 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map
Jun 11 21:15:16 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:16 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 902720 replicas
Jun 11 21:15:16 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:16 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map
Jun 11 21:15:16 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:16 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 902720 replicas
Jun 11 21:15:16 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:16 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map
Jun 11 21:15:17 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:17 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 902720 replicas
Jun 11 21:15:17 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:15:17 mount zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): index done
Jun 11 21:16:24 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:24 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map
Jun 11 21:16:25 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:25 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas
Jun 11 21:16:25 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:25 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map
Jun 11 21:16:25 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:25 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas
Jun 11 21:16:25 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:25 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map
Jun 11 21:16:25 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:25 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas
Jun 11 21:16:26 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:25 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map
Jun 11 21:16:26 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:26 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas
Jun 11 21:16:26 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:26 mount zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): index done
Jun 11 21:16:26 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:26 GetCurrentState: took 10m39.754533229s
Jun 11 21:16:26 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:26 ComputeChangeSets: start
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 ComputeChangeSets: took 26.815637212s
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 ===
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 0 replicas (0 blocks, 0 bytes) lost (0=have<want)
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 0 replicas (0 blocks, 0 bytes) underreplicated (0<have<want)
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 6714780 replicas (3357390 blocks, 398421123119718 bytes) just right (have=want)
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 308 replicas (150 blocks, 2035430788 bytes) overreplicated (have>want>0)
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new)
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old)
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 ===
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 storage class "default": 6715080 replicas (3357540 blocks, 398423158486764 bytes) desired
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 storage class "default": 0 replicas (0 blocks, 0 bytes) short
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 storage class "default": 308 replicas (150 blocks, 2035430788 bytes) surplus
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 storage class "default": 0 replicas (0 blocks, 0 bytes) unachievable
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 ===
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 6715080 replicas (3357540 blocks, 398423158486764 bytes) total commitment (excluding unreferenced)
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 6715388 replicas (3357540 blocks, 398425193917552 bytes) total usage
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 ===
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:0, Trashes:0}
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:0, Trashes:0}
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:0, Trashes:0}
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:0, Trashes:0}
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 ===
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 Replication level distribution (counting N replicas on a single server as N):
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53  0:       0
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53  1:       0
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53  2: 3357390 ###########################################################
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53  3:       0
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53  4:     146 ###################
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53  5:       0
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53  6:       4 ######
Jun 11 21:16:53 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:53 ===
Jun 11 21:16:54 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:54 Run: took 11m8.098097182s
Jun 11 21:16:54 dhhck.arvadosapi.com keep-balance[22156]: 2018/06/11 21:16:54 run succeeded