Actions
Task #13520
closed
Bug #13427: [keep-balance] Handle volumes that are mounted simultaneously by multiple servers
Review 13427-multiple-mounts
Added by Jesse Masterman over 6 years ago. Updated over 6 years ago.
Start date:
06/06/2018
Due date:
% Done:
0%
Estimated time:
Updated by Tom Clegg over 6 years ago
- Subject changed from Review to Review 13427-multiple-mounts
- Status changed from New to In Progress
- Start date set to 06/06/2018
Updated by Ward Vandewege over 6 years ago
keep-balance run prior to this branch:
Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 starting next run Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 Run: start Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 skipping dhhck-bi6l4-blb6dtx0sx07d7k with service type "proxy" Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 GetCurrentState: start Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 zzzzz-ivpuk-aexopqahib74kpp (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com: Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 zzzzz-ivpuk-aj576e1t62use9i (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com: Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 zzzzz-ivpuk-v2udip63fnkdyxf (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com: Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 zzzzz-ivpuk-mkmes11v77mlcso (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com: Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 collections: 0/137060 Jun 06 11:34:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:16 collections: 1000/137060 Jun 06 11:34:17 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:17 collections: 2000/137060 Jun 06 11:34:18 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:18 collections: 3000/137060 Jun 06 11:34:19 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:19 collections: 4000/137060 Jun 06 11:34:20 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:20 collections: 5000/137060 Jun 06 11:34:21 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:21 collections: 6000/137060 Jun 06 11:34:22 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:22 collections: 7000/137060 Jun 06 11:34:23 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:23 collections: 8000/137060 Jun 06 11:34:23 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:23 collections: 9000/137060 Jun 06 11:34:24 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:24 collections: 10000/137060 Jun 06 11:34:25 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:25 collections: 11000/137060 Jun 06 11:34:26 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:26 collections: 12000/137060 Jun 06 11:34:26 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:26 collections: 13000/137060 Jun 06 11:34:27 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:27 collections: 14000/137060 Jun 06 11:34:28 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:28 collections: 15000/137060 Jun 06 11:34:29 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:29 collections: 16000/137060 Jun 06 11:34:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:30 collections: 17000/137060 Jun 06 11:34:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:31 collections: 18000/137060 Jun 06 11:34:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:31 collections: 19000/137060 Jun 06 11:34:33 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:33 collections: 20000/137060 Jun 06 11:34:35 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:35 collections: 21000/137060 Jun 06 11:34:38 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:38 collections: 22000/137060 Jun 06 11:34:39 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:39 collections: 23000/137060 Jun 06 11:34:40 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:40 collections: 24000/137060 Jun 06 11:34:40 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:40 collections: 25000/137060 Jun 06 11:34:41 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:41 collections: 26000/137060 Jun 06 11:34:42 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:42 collections: 27000/137060 Jun 06 11:34:43 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:43 collections: 28000/137060 Jun 06 11:34:44 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:44 collections: 29000/137060 Jun 06 11:34:45 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:45 collections: 30000/137060 Jun 06 11:34:46 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:46 collections: 31000/137060 Jun 06 11:34:47 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:47 collections: 32000/137060 Jun 06 11:34:47 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:47 collections: 33000/137060 Jun 06 11:34:48 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:48 collections: 34000/137060 Jun 06 11:34:49 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:49 collections: 35000/137060 Jun 06 11:34:51 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:51 collections: 36000/137060 Jun 06 11:34:52 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:52 collections: 37000/137060 Jun 06 11:34:52 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:52 collections: 38000/137060 Jun 06 11:34:54 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:54 collections: 39000/137060 Jun 06 11:34:54 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:54 collections: 40000/137060 Jun 06 11:34:55 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:55 collections: 41000/137060 Jun 06 11:34:56 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:56 collections: 42000/137060 Jun 06 11:34:57 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:57 collections: 43000/137060 Jun 06 11:34:58 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:58 collections: 44000/137060 Jun 06 11:34:59 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:34:59 collections: 45000/137060 Jun 06 11:35:00 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:00 collections: 46000/137060 Jun 06 11:35:01 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:01 collections: 47000/137060 Jun 06 11:35:02 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:02 collections: 48000/137060 Jun 06 11:35:03 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:03 collections: 49000/137060 Jun 06 11:35:04 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:04 collections: 50000/137060 Jun 06 11:35:05 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:05 collections: 51000/137060 Jun 06 11:35:05 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:05 collections: 52000/137060 Jun 06 11:35:06 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:06 collections: 53000/137060 Jun 06 11:35:07 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:07 collections: 54000/137060 Jun 06 11:35:08 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:08 collections: 55000/137060 Jun 06 11:35:09 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:09 collections: 56000/137060 Jun 06 11:35:10 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:10 collections: 57000/137060 Jun 06 11:35:11 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:11 collections: 58000/137060 Jun 06 11:35:12 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:12 collections: 59000/137060 Jun 06 11:35:13 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:13 collections: 60000/137060 Jun 06 11:35:14 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:14 collections: 61000/137060 Jun 06 11:35:15 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:15 collections: 62000/137060 Jun 06 11:35:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:16 collections: 63000/137060 Jun 06 11:35:17 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:17 collections: 64000/137060 Jun 06 11:35:18 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:18 collections: 65000/137060 Jun 06 11:35:18 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:18 collections: 66000/137060 Jun 06 11:35:19 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:19 collections: 67000/137060 Jun 06 11:35:20 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:20 collections: 68000/137060 Jun 06 11:35:21 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:21 collections: 69000/137060 Jun 06 11:35:22 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:22 collections: 70000/137060 Jun 06 11:35:23 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:23 collections: 71000/137060 Jun 06 11:35:24 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:24 collections: 72000/137060 Jun 06 11:35:25 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:25 collections: 73000/137060 Jun 06 11:35:26 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:26 collections: 74000/137060 Jun 06 11:35:27 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:27 collections: 75000/137060 Jun 06 11:35:28 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:28 collections: 76000/137060 Jun 06 11:35:29 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:29 collections: 77000/137060 Jun 06 11:35:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:30 collections: 78000/137060 Jun 06 11:35:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:31 collections: 79000/137060 Jun 06 11:35:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:31 collections: 80000/137060 Jun 06 11:35:32 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:32 collections: 81000/137060 Jun 06 11:35:33 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:33 collections: 82000/137060 Jun 06 11:35:34 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:34 collections: 83000/137060 Jun 06 11:35:35 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:35 collections: 84000/137060 Jun 06 11:35:36 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:36 collections: 85000/137060 Jun 06 11:35:37 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:37 collections: 86000/137060 Jun 06 11:35:38 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:38 collections: 87000/137060 Jun 06 11:35:38 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:38 collections: 88000/137060 Jun 06 11:35:39 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:39 collections: 89000/137060 Jun 06 11:35:40 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:40 collections: 90000/137060 Jun 06 11:35:41 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:41 collections: 91000/137060 Jun 06 11:35:42 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:42 collections: 92000/137060 Jun 06 11:35:43 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:43 collections: 93000/137060 Jun 06 11:35:43 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:43 collections: 94000/137060 Jun 06 11:35:44 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:44 collections: 95000/137060 Jun 06 11:35:45 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:45 collections: 96000/137060 Jun 06 11:35:46 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:46 collections: 97000/137060 Jun 06 11:35:46 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:46 collections: 98000/137060 Jun 06 11:35:47 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:47 collections: 99000/137060 Jun 06 11:35:48 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:48 collections: 100000/137060 Jun 06 11:35:49 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:49 collections: 101000/137060 Jun 06 11:35:49 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:49 collections: 102000/137060 Jun 06 11:35:50 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:50 collections: 103000/137060 Jun 06 11:35:51 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:51 collections: 104000/137060 Jun 06 11:35:51 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:51 collections: 105000/137060 Jun 06 11:35:52 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:52 collections: 106000/137060 Jun 06 11:35:53 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:53 collections: 107000/137060 Jun 06 11:35:54 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:54 collections: 108000/137060 Jun 06 11:35:54 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:54 collections: 109000/137060 Jun 06 11:35:55 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:55 collections: 110000/137060 Jun 06 11:35:56 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:56 collections: 111000/137060 Jun 06 11:35:56 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:56 collections: 112000/137060 Jun 06 11:35:57 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:57 collections: 113000/137060 Jun 06 11:35:58 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:58 collections: 114000/137060 Jun 06 11:35:59 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:59 collections: 115000/137060 Jun 06 11:35:59 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:35:59 collections: 116000/137060 Jun 06 11:36:00 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:00 collections: 117000/137060 Jun 06 11:36:00 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:00 collections: 118000/137060 Jun 06 11:36:01 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:01 collections: 119000/137060 Jun 06 11:36:02 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:02 collections: 120000/137060 Jun 06 11:36:03 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:03 collections: 121000/137060 Jun 06 11:36:04 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:04 collections: 122000/137060 Jun 06 11:36:04 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:04 collections: 123000/137060 Jun 06 11:36:05 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:05 collections: 124000/137060 Jun 06 11:36:06 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:06 collections: 125000/137060 Jun 06 11:36:06 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:06 collections: 126000/137060 Jun 06 11:36:07 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:07 collections: 127000/137060 Jun 06 11:36:07 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:07 collections: 128000/137060 Jun 06 11:36:08 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:08 collections: 129000/137060 Jun 06 11:36:09 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:09 collections: 130000/137060 Jun 06 11:36:10 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:10 collections: 131000/137060 Jun 06 11:36:10 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:10 collections: 132000/137060 Jun 06 11:36:11 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:11 collections: 133000/137060 Jun 06 11:36:12 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:12 collections: 134000/137060 Jun 06 11:36:12 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:12 collections: 135000/137060 Jun 06 11:36:14 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:14 collections: 136000/137060 Jun 06 11:36:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:16 collections: 137000/137060 Jun 06 11:36:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:16 collections: 137060/137060 Jun 06 11:36:16 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:36:16 collections: 137060/137060 Jun 06 11:39:14 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:39:14 zzzzz-ivpuk-aexopqahib74kpp (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com: Jun 06 11:39:15 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:39:15 zzzzz-ivpuk-aexopqahib74kpp (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com: Jun 06 11:39:15 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:39:15 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 11:41:43 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:41:43 zzzzz-ivpuk-mkmes11v77mlcso (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com: Jun 06 11:41:43 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:41:43 zzzzz-ivpuk-mkmes11v77mlcso (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com: Jun 06 11:41:43 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:41:43 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 11:41:46 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:41:46 zzzzz-ivpuk-aj576e1t62use9i (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com: Jun 06 11:41:46 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:41:46 zzzzz-ivpuk-aj576e1t62use9i (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com: Jun 06 11:41:46 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:41:46 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 11:43:18 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:18 zzzzz-ivpuk-v2udip63fnkdyxf (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com: Jun 06 11:43:18 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:18 zzzzz-ivpuk-v2udip63fnkdyxf (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com: Jun 06 11:43:18 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:18 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 11:43:18 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:18 GetCurrentState: took 9m2.682677758s Jun 06 11:43:18 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:18 ComputeChangeSets: start Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 ComputeChangeSets: took 11.414222609s Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 === Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 0 replicas (0 blocks, 0 bytes) lost (0=have<want) Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 3357390 replicas (3357390 blocks, 199210561559859 bytes) underreplicated (0<have<want) Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 292 replicas (146 blocks, 2035303304 bytes) just right (have=want) Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 4 replicas (4 blocks, 31871 bytes) overreplicated (have>want>0) Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new) Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old) Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 === Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 storage class "default": 6715080 replicas (3357540 blocks, 398423158486764 bytes) desired Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 storage class "default": 3357390 replicas (3357390 blocks, 199210561559859 bytes) short Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 storage class "default": 4 replicas (4 blocks, 31871 bytes) surplus Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 storage class "default": 0 replicas (0 blocks, 0 bytes) unachievable Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 === Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 6715080 replicas (3357540 blocks, 398423158486764 bytes) total commitment (excluding unreferenced) Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 3357694 replicas (3357540 blocks, 199212596958776 bytes) total usage Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 === Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:777131, Trashes:0} Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:780581, Trashes:0} Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:729884, Trashes:0} Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:1164750, Trashes:0} Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 === Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 Replication level distribution (counting N replicas on a single server as N): Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 0: 0 Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 1: 3357390 ########################################################### Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 2: 146 ################### Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 3: 4 ###### Jun 06 11:43:30 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:30 === Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): send trash list: start Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): send trash list: start Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): send trash list: start Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): send trash list: start Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): send trash list: took 762.937µs Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): send trash list: took 1.386957ms Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): send trash list: took 1.556788ms Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): send trash list: took 2.354539ms Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 Run: took 9m15.607722359s Jun 06 11:43:31 dhhck.arvadosapi.com keep-balance[21398]: 2018/06/06 11:43:31 run succeeded
Updated by Ward Vandewege over 6 years ago
keep-balance run on this branch, without duplicate mounts on the keepstores:
Jun 06 17:40:46 dhhck.arvadosapi.com systemd[1]: Started Arvados Keep Balance. Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 keep-balance dev started Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 starting up: will scan every 4h0m0s and on SIGUSR1 Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 WARNING: Will scan periodically, but no changes will be committed. Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 ======= Consider using -commit-pulls and -commit-trash flags. Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 Run: start Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 skipping dhhck-bi6l4-blb6dtx0sx07d7k with service type "proxy" Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 GetCurrentState: start Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 zzzzz-ivpuk-aj576e1t62use9i (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com: Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 zzzzz-ivpuk-mkmes11v77mlcso (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com: Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 zzzzz-ivpuk-v2udip63fnkdyxf (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com: Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 zzzzz-ivpuk-aexopqahib74kpp (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com: Jun 06 17:40:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:46 collections: 0/137060 Jun 06 17:40:47 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:47 collections: 1000/137060 Jun 06 17:40:48 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:48 collections: 2000/137060 Jun 06 17:40:49 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:49 collections: 3000/137060 Jun 06 17:40:50 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:50 collections: 4000/137060 Jun 06 17:40:51 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:51 collections: 5000/137060 Jun 06 17:40:51 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:51 collections: 6000/137060 Jun 06 17:40:52 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:52 collections: 7000/137060 Jun 06 17:40:53 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:53 collections: 8000/137060 Jun 06 17:40:54 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:54 collections: 9000/137060 Jun 06 17:40:55 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:55 collections: 10000/137060 Jun 06 17:40:55 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:55 collections: 11000/137060 Jun 06 17:40:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:56 collections: 12000/137060 Jun 06 17:40:57 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:57 collections: 13000/137060 Jun 06 17:40:58 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:58 collections: 14000/137060 Jun 06 17:40:59 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:59 collections: 15000/137060 Jun 06 17:40:59 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:40:59 collections: 16000/137060 Jun 06 17:41:00 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:00 collections: 17000/137060 Jun 06 17:41:01 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:01 collections: 18000/137060 Jun 06 17:41:02 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:02 collections: 19000/137060 Jun 06 17:41:04 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:04 collections: 20000/137060 Jun 06 17:41:06 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:06 collections: 21000/137060 Jun 06 17:41:08 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:08 collections: 22000/137060 Jun 06 17:41:09 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:09 collections: 23000/137060 Jun 06 17:41:10 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:10 collections: 24000/137060 Jun 06 17:41:11 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:11 collections: 25000/137060 Jun 06 17:41:12 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:12 collections: 26000/137060 Jun 06 17:41:13 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:13 collections: 27000/137060 Jun 06 17:41:14 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:14 collections: 28000/137060 Jun 06 17:41:14 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:14 collections: 29000/137060 Jun 06 17:41:16 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:16 collections: 30000/137060 Jun 06 17:41:16 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:16 collections: 31000/137060 Jun 06 17:41:17 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:17 collections: 32000/137060 Jun 06 17:41:18 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:18 collections: 33000/137060 Jun 06 17:41:19 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:19 collections: 34000/137060 Jun 06 17:41:20 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:20 collections: 35000/137060 Jun 06 17:41:21 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:21 collections: 36000/137060 Jun 06 17:41:22 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:22 collections: 37000/137060 Jun 06 17:41:23 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:23 collections: 38000/137060 Jun 06 17:41:24 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:24 collections: 39000/137060 Jun 06 17:41:25 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:25 collections: 40000/137060 Jun 06 17:41:26 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:26 collections: 41000/137060 Jun 06 17:41:27 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:27 collections: 42000/137060 Jun 06 17:41:28 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:28 collections: 43000/137060 Jun 06 17:41:29 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:29 collections: 44000/137060 Jun 06 17:41:30 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:30 collections: 45000/137060 Jun 06 17:41:30 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:30 collections: 46000/137060 Jun 06 17:41:31 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:31 collections: 47000/137060 Jun 06 17:41:32 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:32 collections: 48000/137060 Jun 06 17:41:33 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:33 collections: 49000/137060 Jun 06 17:41:34 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:34 collections: 50000/137060 Jun 06 17:41:35 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:35 collections: 51000/137060 Jun 06 17:41:36 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:36 collections: 52000/137060 Jun 06 17:41:37 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:37 collections: 53000/137060 Jun 06 17:41:38 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:38 collections: 54000/137060 Jun 06 17:41:39 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:39 collections: 55000/137060 Jun 06 17:41:40 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:40 collections: 56000/137060 Jun 06 17:41:40 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:40 collections: 57000/137060 Jun 06 17:41:41 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:41 collections: 58000/137060 Jun 06 17:41:42 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:42 collections: 59000/137060 Jun 06 17:41:43 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:43 collections: 60000/137060 Jun 06 17:41:44 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:44 collections: 61000/137060 Jun 06 17:41:45 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:45 collections: 62000/137060 Jun 06 17:41:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:46 collections: 63000/137060 Jun 06 17:41:47 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:47 collections: 64000/137060 Jun 06 17:41:48 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:48 collections: 65000/137060 Jun 06 17:41:49 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:49 collections: 66000/137060 Jun 06 17:41:50 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:50 collections: 67000/137060 Jun 06 17:41:51 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:51 collections: 68000/137060 Jun 06 17:41:52 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:52 collections: 69000/137060 Jun 06 17:41:53 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:53 collections: 70000/137060 Jun 06 17:41:54 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:54 collections: 71000/137060 Jun 06 17:41:55 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:55 collections: 72000/137060 Jun 06 17:41:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:56 collections: 73000/137060 Jun 06 17:41:57 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:57 collections: 74000/137060 Jun 06 17:41:58 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:58 collections: 75000/137060 Jun 06 17:41:59 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:41:59 collections: 76000/137060 Jun 06 17:42:00 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:00 collections: 77000/137060 Jun 06 17:42:00 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:00 collections: 78000/137060 Jun 06 17:42:01 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:01 collections: 79000/137060 Jun 06 17:42:02 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:02 collections: 80000/137060 Jun 06 17:42:03 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:03 collections: 81000/137060 Jun 06 17:42:04 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:04 collections: 82000/137060 Jun 06 17:42:05 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:05 collections: 83000/137060 Jun 06 17:42:06 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:06 collections: 84000/137060 Jun 06 17:42:07 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:07 collections: 85000/137060 Jun 06 17:42:07 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:07 collections: 86000/137060 Jun 06 17:42:08 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:08 collections: 87000/137060 Jun 06 17:42:09 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:09 collections: 88000/137060 Jun 06 17:42:10 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:10 collections: 89000/137060 Jun 06 17:42:10 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:10 collections: 90000/137060 Jun 06 17:42:12 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:12 collections: 91000/137060 Jun 06 17:42:12 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:12 collections: 92000/137060 Jun 06 17:42:13 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:13 collections: 93000/137060 Jun 06 17:42:14 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:14 collections: 94000/137060 Jun 06 17:42:15 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:15 collections: 95000/137060 Jun 06 17:42:15 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:15 collections: 96000/137060 Jun 06 17:42:16 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:16 collections: 97000/137060 Jun 06 17:42:17 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:17 collections: 98000/137060 Jun 06 17:42:18 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:18 collections: 99000/137060 Jun 06 17:42:18 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:18 collections: 100000/137060 Jun 06 17:42:19 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:19 collections: 101000/137060 Jun 06 17:42:20 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:20 collections: 102000/137060 Jun 06 17:42:21 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:21 collections: 103000/137060 Jun 06 17:42:21 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:21 collections: 104000/137060 Jun 06 17:42:22 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:22 collections: 105000/137060 Jun 06 17:42:23 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:23 collections: 106000/137060 Jun 06 17:42:23 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:23 collections: 107000/137060 Jun 06 17:42:24 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:24 collections: 108000/137060 Jun 06 17:42:25 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:25 collections: 109000/137060 Jun 06 17:42:25 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:25 collections: 110000/137060 Jun 06 17:42:26 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:26 collections: 111000/137060 Jun 06 17:42:27 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:27 collections: 112000/137060 Jun 06 17:42:28 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:28 collections: 113000/137060 Jun 06 17:42:28 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:28 collections: 114000/137060 Jun 06 17:42:29 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:29 collections: 115000/137060 Jun 06 17:42:30 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:30 collections: 116000/137060 Jun 06 17:42:31 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:31 collections: 117000/137060 Jun 06 17:42:31 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:31 collections: 118000/137060 Jun 06 17:42:32 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:32 collections: 119000/137060 Jun 06 17:42:33 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:33 collections: 120000/137060 Jun 06 17:42:34 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:34 collections: 121000/137060 Jun 06 17:42:34 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:34 collections: 122000/137060 Jun 06 17:42:35 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:35 collections: 123000/137060 Jun 06 17:42:36 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:36 collections: 124000/137060 Jun 06 17:42:36 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:36 collections: 125000/137060 Jun 06 17:42:37 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:37 collections: 126000/137060 Jun 06 17:42:38 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:38 collections: 127000/137060 Jun 06 17:42:38 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:38 collections: 128000/137060 Jun 06 17:42:39 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:39 collections: 129000/137060 Jun 06 17:42:40 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:40 collections: 130000/137060 Jun 06 17:42:41 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:41 collections: 131000/137060 Jun 06 17:42:41 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:41 collections: 132000/137060 Jun 06 17:42:42 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:42 collections: 133000/137060 Jun 06 17:42:42 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:42 collections: 134000/137060 Jun 06 17:42:43 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:43 collections: 135000/137060 Jun 06 17:42:44 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:44 collections: 136000/137060 Jun 06 17:42:46 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:46 collections: 137000/137060 Jun 06 17:42:47 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:47 collections: 137060/137060 Jun 06 17:42:47 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:42:47 collections: 137060/137060 Jun 06 17:46:43 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:46:43 zzzzz-ivpuk-aexopqahib74kpp (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com: Jun 06 17:46:43 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:46:43 zzzzz-ivpuk-aexopqahib74kpp (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com: Jun 06 17:46:43 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:46:43 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 17:49:32 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:49:32 zzzzz-ivpuk-aj576e1t62use9i (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com: Jun 06 17:49:33 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:49:33 zzzzz-ivpuk-mkmes11v77mlcso (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com: Jun 06 17:49:33 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:49:33 zzzzz-ivpuk-aj576e1t62use9i (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com: Jun 06 17:49:33 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:49:33 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 17:49:33 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:49:33 zzzzz-ivpuk-mkmes11v77mlcso (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com: Jun 06 17:49:33 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:49:33 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 17:50:43 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:43 zzzzz-ivpuk-v2udip63fnkdyxf (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com: Jun 06 17:50:44 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:44 zzzzz-ivpuk-v2udip63fnkdyxf (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com: Jun 06 17:50:44 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:44 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 17:50:44 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:44 GetCurrentState: took 9m57.513241214s Jun 06 17:50:44 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:44 ComputeChangeSets: start Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 ComputeChangeSets: took 11.761291821s Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 === Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 0 replicas (0 blocks, 0 bytes) lost (0=have<want) Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 3357390 replicas (3357390 blocks, 199210561559859 bytes) underreplicated (0<have<want) Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 292 replicas (146 blocks, 2035303304 bytes) just right (have=want) Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 4 replicas (4 blocks, 31871 bytes) overreplicated (have>want>0) Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new) Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old) Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 === Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 storage class "default": 6715080 replicas (3357540 blocks, 398423158486764 bytes) desired Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 storage class "default": 3357390 replicas (3357390 blocks, 199210561559859 bytes) short Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 storage class "default": 4 replicas (4 blocks, 31871 bytes) surplus Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 storage class "default": 0 replicas (0 blocks, 0 bytes) unachievable Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 === Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 6715080 replicas (3357540 blocks, 398423158486764 bytes) total commitment (excluding unreferenced) Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 3357694 replicas (3357540 blocks, 199212596958776 bytes) total usage Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 === Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:780581, Trashes:0} Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:729884, Trashes:0} Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:1164750, Trashes:0} Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:777131, Trashes:0} Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 === Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 Replication level distribution (counting N replicas on a single server as N): Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 0: 0 Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 1: 3357390 ########################################################### Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 2: 146 ################### Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 3: 4 ###### Jun 06 17:50:56 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:56 === Jun 06 17:50:57 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:57 Run: took 10m10.792658967s Jun 06 17:50:57 dhhck.arvadosapi.com keep-balance[29018]: 2018/06/06 17:50:57 run succeeded
Updated by Ward Vandewege over 6 years ago
keep-balance run on this branch, with duplicate mounts on the keepstores (one rw, all the others ro):
Jun 06 17:57:24 dhhck.arvadosapi.com systemd[1]: Started Arvados Keep Balance. Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 keep-balance dev started Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 starting up: will scan every 4h0m0s and on SIGUSR1 Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 WARNING: Will scan periodically, but no changes will be committed. Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 ======= Consider using -commit-pulls and -commit-trash flags. Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 Run: start Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping dhhck-bi6l4-blb6dtx0sx07d7k with service type "proxy" Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-b4om76522uxf6q8" because same device "s3:///dhhck-keep-2" is mounted read-write on srv dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-8ga3p0bfixahtbx" because same device "s3:///dhhck-keep-0" is mounted read-write on srv dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-jb3gqert37cyqer" because same device "s3:///dhhck-keep-1" is mounted read-write on srv dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-uasul11r1ot5042" because same device "s3:///dhhck-keep-3" is mounted read-write on srv dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-zqcxugw4iuym4o3" because same device "s3:///dhhck-keep-0" is mounted read-write on srv dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-qpp0g6xhamdtpml" because same device "s3:///dhhck-keep-1" is mounted read-write on srv dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-0x3xs3orqugcrrl" because same device "s3:///dhhck-keep-1" is mounted read-write on srv dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-uhq59w5uuon0tp7" because same device "s3:///dhhck-keep-2" is mounted read-write on srv dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-phxzrhwvogz9kjf" because same device "s3:///dhhck-keep-3" is mounted read-write on srv dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-asgutjzipuem6tj" because same device "s3:///dhhck-keep-0" is mounted read-write on srv dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-nodsuucvm60ks5p" because same device "s3:///dhhck-keep-2" is mounted read-write on srv dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 skipping srv dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3) readonly mount "zzzzz-ivpuk-4d1is7hgq2x4kp5" because same device "s3:///dhhck-keep-3" is mounted read-write on srv dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvado Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 GetCurrentState: start Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 zzzzz-ivpuk-xnlmcuarjtmjint (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 zzzzz-ivpuk-yzdywmvvfh91fvm (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 zzzzz-ivpuk-f7g5pgtr00229nb (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 zzzzz-ivpuk-h2pk4r82lrmrz8a (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 17:57:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:24 collections: 0/137060 Jun 06 17:57:25 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:25 collections: 1000/137060 Jun 06 17:57:26 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:26 collections: 2000/137060 Jun 06 17:57:27 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:27 collections: 3000/137060 Jun 06 17:57:27 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:27 collections: 4000/137060 Jun 06 17:57:29 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:29 collections: 5000/137060 Jun 06 17:57:29 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:29 collections: 6000/137060 Jun 06 17:57:30 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:30 collections: 7000/137060 Jun 06 17:57:31 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:31 collections: 8000/137060 Jun 06 17:57:32 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:32 collections: 9000/137060 Jun 06 17:57:32 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:32 collections: 10000/137060 Jun 06 17:57:33 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:33 collections: 11000/137060 Jun 06 17:57:34 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:34 collections: 12000/137060 Jun 06 17:57:35 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:35 collections: 13000/137060 Jun 06 17:57:36 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:36 collections: 14000/137060 Jun 06 17:57:36 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:36 collections: 15000/137060 Jun 06 17:57:37 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:37 collections: 16000/137060 Jun 06 17:57:38 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:38 collections: 17000/137060 Jun 06 17:57:39 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:39 collections: 18000/137060 Jun 06 17:57:40 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:40 collections: 19000/137060 Jun 06 17:57:41 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:41 collections: 20000/137060 Jun 06 17:57:43 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:43 collections: 21000/137060 Jun 06 17:57:46 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:46 collections: 22000/137060 Jun 06 17:57:47 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:47 collections: 23000/137060 Jun 06 17:57:47 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:47 collections: 24000/137060 Jun 06 17:57:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:48 collections: 25000/137060 Jun 06 17:57:49 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:49 collections: 26000/137060 Jun 06 17:57:50 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:50 collections: 27000/137060 Jun 06 17:57:51 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:51 collections: 28000/137060 Jun 06 17:57:52 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:52 collections: 29000/137060 Jun 06 17:57:53 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:53 collections: 30000/137060 Jun 06 17:57:54 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:54 collections: 31000/137060 Jun 06 17:57:55 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:55 collections: 32000/137060 Jun 06 17:57:56 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:56 collections: 33000/137060 Jun 06 17:57:56 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:56 collections: 34000/137060 Jun 06 17:57:57 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:57 collections: 35000/137060 Jun 06 17:57:58 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:58 collections: 36000/137060 Jun 06 17:57:59 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:57:59 collections: 37000/137060 Jun 06 17:58:00 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:00 collections: 38000/137060 Jun 06 17:58:01 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:01 collections: 39000/137060 Jun 06 17:58:02 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:02 collections: 40000/137060 Jun 06 17:58:03 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:03 collections: 41000/137060 Jun 06 17:58:04 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:04 collections: 42000/137060 Jun 06 17:58:05 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:05 collections: 43000/137060 Jun 06 17:58:06 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:06 collections: 44000/137060 Jun 06 17:58:07 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:07 collections: 45000/137060 Jun 06 17:58:08 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:08 collections: 46000/137060 Jun 06 17:58:09 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:09 collections: 47000/137060 Jun 06 17:58:10 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:10 collections: 48000/137060 Jun 06 17:58:10 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:10 collections: 49000/137060 Jun 06 17:58:11 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:11 collections: 50000/137060 Jun 06 17:58:12 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:12 collections: 51000/137060 Jun 06 17:58:13 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:13 collections: 52000/137060 Jun 06 17:58:14 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:14 collections: 53000/137060 Jun 06 17:58:15 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:15 collections: 54000/137060 Jun 06 17:58:16 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:16 collections: 55000/137060 Jun 06 17:58:17 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:17 collections: 56000/137060 Jun 06 17:58:18 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:18 collections: 57000/137060 Jun 06 17:58:19 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:19 collections: 58000/137060 Jun 06 17:58:20 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:20 collections: 59000/137060 Jun 06 17:58:21 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:21 collections: 60000/137060 Jun 06 17:58:22 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:22 collections: 61000/137060 Jun 06 17:58:23 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:23 collections: 62000/137060 Jun 06 17:58:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:24 collections: 63000/137060 Jun 06 17:58:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:24 collections: 64000/137060 Jun 06 17:58:25 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:25 collections: 65000/137060 Jun 06 17:58:26 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:26 collections: 66000/137060 Jun 06 17:58:27 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:27 collections: 67000/137060 Jun 06 17:58:28 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:28 collections: 68000/137060 Jun 06 17:58:29 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:29 collections: 69000/137060 Jun 06 17:58:30 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:30 collections: 70000/137060 Jun 06 17:58:31 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:31 collections: 71000/137060 Jun 06 17:58:32 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:32 collections: 72000/137060 Jun 06 17:58:33 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:33 collections: 73000/137060 Jun 06 17:58:34 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:34 collections: 74000/137060 Jun 06 17:58:35 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:35 collections: 75000/137060 Jun 06 17:58:35 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:35 collections: 76000/137060 Jun 06 17:58:36 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:36 collections: 77000/137060 Jun 06 17:58:37 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:37 collections: 78000/137060 Jun 06 17:58:38 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:38 collections: 79000/137060 Jun 06 17:58:39 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:39 collections: 80000/137060 Jun 06 17:58:40 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:40 collections: 81000/137060 Jun 06 17:58:41 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:41 collections: 82000/137060 Jun 06 17:58:42 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:42 collections: 83000/137060 Jun 06 17:58:43 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:43 collections: 84000/137060 Jun 06 17:58:44 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:44 collections: 85000/137060 Jun 06 17:58:44 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:44 collections: 86000/137060 Jun 06 17:58:45 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:45 collections: 87000/137060 Jun 06 17:58:46 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:46 collections: 88000/137060 Jun 06 17:58:47 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:47 collections: 89000/137060 Jun 06 17:58:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:48 collections: 90000/137060 Jun 06 17:58:49 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:49 collections: 91000/137060 Jun 06 17:58:50 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:50 collections: 92000/137060 Jun 06 17:58:51 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:51 collections: 93000/137060 Jun 06 17:58:51 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:51 collections: 94000/137060 Jun 06 17:58:52 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:52 collections: 95000/137060 Jun 06 17:58:53 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:53 collections: 96000/137060 Jun 06 17:58:54 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:54 collections: 97000/137060 Jun 06 17:58:54 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:54 collections: 98000/137060 Jun 06 17:58:55 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:55 collections: 99000/137060 Jun 06 17:58:56 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:56 collections: 100000/137060 Jun 06 17:58:57 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:57 collections: 101000/137060 Jun 06 17:58:57 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:57 collections: 102000/137060 Jun 06 17:58:58 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:58 collections: 103000/137060 Jun 06 17:58:59 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:58:59 collections: 104000/137060 Jun 06 17:59:00 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:00 collections: 105000/137060 Jun 06 17:59:00 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:00 collections: 106000/137060 Jun 06 17:59:01 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:01 collections: 107000/137060 Jun 06 17:59:01 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:01 collections: 108000/137060 Jun 06 17:59:02 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:02 collections: 109000/137060 Jun 06 17:59:03 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:03 collections: 110000/137060 Jun 06 17:59:04 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:04 collections: 111000/137060 Jun 06 17:59:04 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:04 collections: 112000/137060 Jun 06 17:59:05 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:05 collections: 113000/137060 Jun 06 17:59:06 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:06 collections: 114000/137060 Jun 06 17:59:07 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:07 collections: 115000/137060 Jun 06 17:59:07 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:07 collections: 116000/137060 Jun 06 17:59:08 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:08 collections: 117000/137060 Jun 06 17:59:09 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:09 collections: 118000/137060 Jun 06 17:59:09 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:09 collections: 119000/137060 Jun 06 17:59:10 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:10 collections: 120000/137060 Jun 06 17:59:11 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:11 collections: 121000/137060 Jun 06 17:59:11 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:11 collections: 122000/137060 Jun 06 17:59:12 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:12 collections: 123000/137060 Jun 06 17:59:13 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:13 collections: 124000/137060 Jun 06 17:59:14 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:14 collections: 125000/137060 Jun 06 17:59:14 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:14 collections: 126000/137060 Jun 06 17:59:15 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:15 collections: 127000/137060 Jun 06 17:59:16 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:16 collections: 128000/137060 Jun 06 17:59:16 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:16 collections: 129000/137060 Jun 06 17:59:17 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:17 collections: 130000/137060 Jun 06 17:59:18 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:18 collections: 131000/137060 Jun 06 17:59:18 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:18 collections: 132000/137060 Jun 06 17:59:19 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:19 collections: 133000/137060 Jun 06 17:59:20 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:20 collections: 134000/137060 Jun 06 17:59:21 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:21 collections: 135000/137060 Jun 06 17:59:22 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:22 collections: 136000/137060 Jun 06 17:59:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:24 collections: 137000/137060 Jun 06 17:59:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:24 collections: 137060/137060 Jun 06 17:59:24 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 17:59:24 collections: 137060/137060 Jun 06 18:03:14 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:03:14 zzzzz-ivpuk-h2pk4r82lrmrz8a (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 18:03:14 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:03:14 zzzzz-ivpuk-h2pk4r82lrmrz8a (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:03:14 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:03:14 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:05:55 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:05:55 zzzzz-ivpuk-xnlmcuarjtmjint (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 18:05:56 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:05:56 zzzzz-ivpuk-xnlmcuarjtmjint (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:05:56 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:05:56 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:06:03 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:06:03 zzzzz-ivpuk-f7g5pgtr00229nb (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 18:06:03 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:06:03 zzzzz-ivpuk-f7g5pgtr00229nb (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:06:03 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:06:03 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:07:36 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:36 zzzzz-ivpuk-yzdywmvvfh91fvm (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 18:07:36 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:36 zzzzz-ivpuk-yzdywmvvfh91fvm (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:07:36 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:36 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:07:36 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:36 GetCurrentState: took 10m12.10938499s Jun 06 18:07:36 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:36 ComputeChangeSets: start Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 ComputeChangeSets: took 11.660132416s Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 === Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 0 replicas (0 blocks, 0 bytes) lost (0=have<want) Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 3357390 replicas (3357390 blocks, 199210561559859 bytes) underreplicated (0<have<want) Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 292 replicas (146 blocks, 2035303304 bytes) just right (have=want) Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 4 replicas (4 blocks, 31871 bytes) overreplicated (have>want>0) Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new) Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old) Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 === Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 storage class "default": 6715080 replicas (3357540 blocks, 398423158486764 bytes) desired Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 storage class "default": 3357390 replicas (3357390 blocks, 199210561559859 bytes) short Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 storage class "default": 4 replicas (4 blocks, 31871 bytes) surplus Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 storage class "default": 0 replicas (0 blocks, 0 bytes) unachievable Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 === Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 6715080 replicas (3357540 blocks, 398423158486764 bytes) total commitment (excluding unreferenced) Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 3357694 replicas (3357540 blocks, 199212596958776 bytes) total usage Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 === Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:777131, Trashes:0} Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:780581, Trashes:0} Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:729884, Trashes:0} Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:1164750, Trashes:0} Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 === Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 Replication level distribution (counting N replicas on a single server as N): Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 0: 0 Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 1: 3357390 ########################################################### Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 2: 146 ################### Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 3: 4 ###### Jun 06 18:07:48 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:48 === Jun 06 18:07:49 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:49 Run: took 10m25.267507844s Jun 06 18:07:49 dhhck.arvadosapi.com keep-balance[835]: 2018/06/06 18:07:49 run succeeded
Updated by Ward Vandewege over 6 years ago
keep-balance run on this branch, with duplicate mounts on the keepstores (all mounts rw on all keepstores):
Jun 06 18:25:57 dhhck.arvadosapi.com systemd[1]: Started Arvados Keep Balance. Jun 06 18:25:57 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:57 keep-balance dev started Jun 06 18:25:57 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:57 starting up: will scan every 4h0m0s and on SIGUSR1 Jun 06 18:25:57 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:57 WARNING: Will scan periodically, but no changes will be committed. Jun 06 18:25:57 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:57 ======= Consider using -commit-pulls and -commit-trash flags. Jun 06 18:25:57 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:57 Run: start Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 skipping dhhck-bi6l4-blb6dtx0sx07d7k with service type "proxy" Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 GetCurrentState: start Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:25:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:58 collections: 0/137060 Jun 06 18:25:59 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:25:59 collections: 1000/137060 Jun 06 18:26:00 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:00 collections: 2000/137060 Jun 06 18:26:01 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:01 collections: 3000/137060 Jun 06 18:26:02 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:02 collections: 4000/137060 Jun 06 18:26:03 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:03 collections: 5000/137060 Jun 06 18:26:04 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:04 collections: 6000/137060 Jun 06 18:26:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:05 collections: 7000/137060 Jun 06 18:26:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:05 collections: 8000/137060 Jun 06 18:26:06 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:06 collections: 9000/137060 Jun 06 18:26:07 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:07 collections: 10000/137060 Jun 06 18:26:08 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:08 collections: 11000/137060 Jun 06 18:26:09 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:09 collections: 12000/137060 Jun 06 18:26:09 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:09 collections: 13000/137060 Jun 06 18:26:10 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:10 collections: 14000/137060 Jun 06 18:26:11 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:11 collections: 15000/137060 Jun 06 18:26:12 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:12 collections: 16000/137060 Jun 06 18:26:13 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:13 collections: 17000/137060 Jun 06 18:26:13 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:13 collections: 18000/137060 Jun 06 18:26:14 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:14 collections: 19000/137060 Jun 06 18:26:16 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:16 collections: 20000/137060 Jun 06 18:26:18 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:18 collections: 21000/137060 Jun 06 18:26:20 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:20 collections: 22000/137060 Jun 06 18:26:21 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:21 collections: 23000/137060 Jun 06 18:26:22 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:22 collections: 24000/137060 Jun 06 18:26:23 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:23 collections: 25000/137060 Jun 06 18:26:24 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:24 collections: 26000/137060 Jun 06 18:26:25 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:25 collections: 27000/137060 Jun 06 18:26:26 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:26 collections: 28000/137060 Jun 06 18:26:26 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:26 collections: 29000/137060 Jun 06 18:26:27 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:27 collections: 30000/137060 Jun 06 18:26:28 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:28 collections: 31000/137060 Jun 06 18:26:29 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:29 collections: 32000/137060 Jun 06 18:26:30 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:30 collections: 33000/137060 Jun 06 18:26:31 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:31 collections: 34000/137060 Jun 06 18:26:32 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:32 collections: 35000/137060 Jun 06 18:26:33 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:33 collections: 36000/137060 Jun 06 18:26:34 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:34 collections: 37000/137060 Jun 06 18:26:35 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:35 collections: 38000/137060 Jun 06 18:26:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:36 collections: 39000/137060 Jun 06 18:26:37 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:37 collections: 40000/137060 Jun 06 18:26:38 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:38 collections: 41000/137060 Jun 06 18:26:39 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:39 collections: 42000/137060 Jun 06 18:26:40 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:40 collections: 43000/137060 Jun 06 18:26:41 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:41 collections: 44000/137060 Jun 06 18:26:42 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:42 collections: 45000/137060 Jun 06 18:26:42 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:42 collections: 46000/137060 Jun 06 18:26:43 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:43 collections: 47000/137060 Jun 06 18:26:44 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:44 collections: 48000/137060 Jun 06 18:26:45 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:45 collections: 49000/137060 Jun 06 18:26:46 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:46 collections: 50000/137060 Jun 06 18:26:47 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:47 collections: 51000/137060 Jun 06 18:26:48 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:48 collections: 52000/137060 Jun 06 18:26:49 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:49 collections: 53000/137060 Jun 06 18:26:50 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:50 collections: 54000/137060 Jun 06 18:26:51 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:51 collections: 55000/137060 Jun 06 18:26:52 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:52 collections: 56000/137060 Jun 06 18:26:53 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:53 collections: 57000/137060 Jun 06 18:26:53 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:53 collections: 58000/137060 Jun 06 18:26:54 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:54 collections: 59000/137060 Jun 06 18:26:55 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:55 collections: 60000/137060 Jun 06 18:26:56 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:56 collections: 61000/137060 Jun 06 18:26:57 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:57 collections: 62000/137060 Jun 06 18:26:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:58 collections: 63000/137060 Jun 06 18:26:59 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:26:59 collections: 64000/137060 Jun 06 18:27:00 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:00 collections: 65000/137060 Jun 06 18:27:01 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:01 collections: 66000/137060 Jun 06 18:27:02 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:02 collections: 67000/137060 Jun 06 18:27:03 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:03 collections: 68000/137060 Jun 06 18:27:04 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:04 collections: 69000/137060 Jun 06 18:27:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:05 collections: 70000/137060 Jun 06 18:27:06 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:06 collections: 71000/137060 Jun 06 18:27:07 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:07 collections: 72000/137060 Jun 06 18:27:08 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:08 collections: 73000/137060 Jun 06 18:27:09 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:09 collections: 74000/137060 Jun 06 18:27:10 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:10 collections: 75000/137060 Jun 06 18:27:11 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:11 collections: 76000/137060 Jun 06 18:27:12 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:12 collections: 77000/137060 Jun 06 18:27:13 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:13 collections: 78000/137060 Jun 06 18:27:13 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:13 collections: 79000/137060 Jun 06 18:27:14 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:14 collections: 80000/137060 Jun 06 18:27:15 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:15 collections: 81000/137060 Jun 06 18:27:16 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:16 collections: 82000/137060 Jun 06 18:27:17 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:17 collections: 83000/137060 Jun 06 18:27:18 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:18 collections: 84000/137060 Jun 06 18:27:19 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:19 collections: 85000/137060 Jun 06 18:27:20 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:20 collections: 86000/137060 Jun 06 18:27:21 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:21 collections: 87000/137060 Jun 06 18:27:21 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:21 collections: 88000/137060 Jun 06 18:27:22 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:22 collections: 89000/137060 Jun 06 18:27:23 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:23 collections: 90000/137060 Jun 06 18:27:25 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:25 collections: 91000/137060 Jun 06 18:27:26 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:26 collections: 92000/137060 Jun 06 18:27:27 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:27 collections: 93000/137060 Jun 06 18:27:27 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:27 collections: 94000/137060 Jun 06 18:27:28 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:28 collections: 95000/137060 Jun 06 18:27:29 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:29 collections: 96000/137060 Jun 06 18:27:30 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:30 collections: 97000/137060 Jun 06 18:27:30 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:30 collections: 98000/137060 Jun 06 18:27:31 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:31 collections: 99000/137060 Jun 06 18:27:32 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:32 collections: 100000/137060 Jun 06 18:27:33 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:33 collections: 101000/137060 Jun 06 18:27:33 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:33 collections: 102000/137060 Jun 06 18:27:34 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:34 collections: 103000/137060 Jun 06 18:27:35 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:35 collections: 104000/137060 Jun 06 18:27:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:36 collections: 105000/137060 Jun 06 18:27:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:36 collections: 106000/137060 Jun 06 18:27:37 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:37 collections: 107000/137060 Jun 06 18:27:38 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:38 collections: 108000/137060 Jun 06 18:27:38 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:38 collections: 109000/137060 Jun 06 18:27:39 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:39 collections: 110000/137060 Jun 06 18:27:40 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:40 collections: 111000/137060 Jun 06 18:27:41 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:41 collections: 112000/137060 Jun 06 18:27:41 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:41 collections: 113000/137060 Jun 06 18:27:42 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:42 collections: 114000/137060 Jun 06 18:27:43 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:43 collections: 115000/137060 Jun 06 18:27:43 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:43 collections: 116000/137060 Jun 06 18:27:44 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:44 collections: 117000/137060 Jun 06 18:27:45 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:45 collections: 118000/137060 Jun 06 18:27:46 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:46 collections: 119000/137060 Jun 06 18:27:46 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:46 collections: 120000/137060 Jun 06 18:27:47 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:47 collections: 121000/137060 Jun 06 18:27:48 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:48 collections: 122000/137060 Jun 06 18:27:49 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:49 collections: 123000/137060 Jun 06 18:27:49 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:49 collections: 124000/137060 Jun 06 18:27:50 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:50 collections: 125000/137060 Jun 06 18:27:51 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:51 collections: 126000/137060 Jun 06 18:27:51 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:51 collections: 127000/137060 Jun 06 18:27:52 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:52 collections: 128000/137060 Jun 06 18:27:53 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:53 collections: 129000/137060 Jun 06 18:27:54 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:54 collections: 130000/137060 Jun 06 18:27:54 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:54 collections: 131000/137060 Jun 06 18:27:55 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:55 collections: 132000/137060 Jun 06 18:27:56 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:56 collections: 133000/137060 Jun 06 18:27:56 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:56 collections: 134000/137060 Jun 06 18:27:57 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:57 collections: 135000/137060 Jun 06 18:27:58 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:27:58 collections: 136000/137060 Jun 06 18:28:01 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:28:01 collections: 137000/137060 Jun 06 18:28:01 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:28:01 collections: 137060/137060 Jun 06 18:28:01 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:28:01 collections: 137060/137060 Jun 06 18:31:41 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:31:41 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 18:31:41 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:31:41 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:31:41 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:31:41 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:34:20 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:34:20 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 18:34:21 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:34:21 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:34:21 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:34:21 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:34:23 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:34:23 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 18:34:23 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:34:23 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:34:23 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:34:23 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:35:48 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:35:48 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 18:35:48 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:35:48 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:35:48 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:35:48 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:41:46 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:41:46 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 18:41:46 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:41:46 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:41:46 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:41:46 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:41:48 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:41:48 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 18:41:48 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:41:48 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:41:48 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:41:48 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:42:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:42:36 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 18:42:37 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:42:37 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:42:37 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:42:37 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:42:54 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:42:54 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 18:42:55 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:42:55 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:42:55 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:42:55 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:49:54 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:49:54 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 18:49:55 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:49:55 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:49:55 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:49:55 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:50:02 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:50:02 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 18:50:02 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:50:02 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:50:02 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:50:02 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:52:18 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:52:18 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 18:52:18 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:52:18 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:52:18 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:52:18 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:52:42 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:52:42 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 18:52:43 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:52:43 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:52:43 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:52:43 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 18:57:54 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:57:54 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 18:57:55 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:57:55 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:57:55 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:57:55 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:58:04 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:04 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 18:58:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:05 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:58:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:05 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:58:25 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:25 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 18:58:25 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:25 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:58:25 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:25 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:58:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:36 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 18:58:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:36 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:58:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:36 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 18:58:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:36 GetCurrentState: took 32m38.257385851s Jun 06 18:58:36 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:58:36 ComputeChangeSets: start Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 ComputeChangeSets: took 28.597287657s Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 === Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 0 replicas (0 blocks, 0 bytes) lost (0=have<want) Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 0 replicas (0 blocks, 0 bytes) underreplicated (0<have<want) Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 0 replicas (0 blocks, 0 bytes) just right (have=want) Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 6715696 replicas (3357540 blocks, 398427229348340 bytes) overreplicated (have>want>0) Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new) Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old) Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 === Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 storage class "default": 6715080 replicas (3357540 blocks, 398423158486764 bytes) desired Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 storage class "default": 0 replicas (0 blocks, 0 bytes) short Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 storage class "default": 6715696 replicas (3357540 blocks, 398427229348340 bytes) surplus Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 storage class "default": 0 replicas (0 blocks, 0 bytes) unachievable Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 === Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 6715080 replicas (3357540 blocks, 398423158486764 bytes) total commitment (excluding unreferenced) Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 13430776 replicas (3357540 blocks, 796850387835104 bytes) total usage Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 === Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:840514, Trashes:0} Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:839577, Trashes:0} Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:838185, Trashes:0} Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:839114, Trashes:0} Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 === Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 Replication level distribution (counting N replicas on a single server as N): Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 0: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 1: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 2: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 3: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 4: 3357390 ########################################################### Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 5: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 6: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 7: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 8: 146 ################### Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 9: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 10: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 11: 0 Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 12: 4 ###### Jun 06 18:59:05 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:05 === Jun 06 18:59:06 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:06 Run: took 33m8.635860787s Jun 06 18:59:06 dhhck.arvadosapi.com keep-balance[8298]: 2018/06/06 18:59:06 run succeeded
Updated by Ward Vandewege over 6 years ago
Testing new 1714431955ac7b2a1d2fd3c2cda9b806bad624cb
keep-balance run on this branch, with duplicate mounts on the keepstores (all mounts rw on all keepstores):
Jun 06 19:31:29 dhhck.arvadosapi.com systemd[1]: Started Arvados Keep Balance. Jun 06 19:31:29 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:29 keep-balance dev started Jun 06 19:31:29 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:29 starting up: will scan every 4h0m0s and on SIGUSR1 Jun 06 19:31:29 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:29 WARNING: Will scan periodically, but no changes will be committed. Jun 06 19:31:29 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:29 ======= Consider using -commit-pulls and -commit-trash flags. Jun 06 19:31:29 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:29 Run: start Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 skipping dhhck-bi6l4-blb6dtx0sx07d7k with service type "proxy" Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 GetCurrentState: start Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve indexes Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 collections: 0/137060 Jun 06 19:31:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:30 collections: 1000/137060 Jun 06 19:31:31 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:31 collections: 2000/137060 Jun 06 19:31:32 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:32 collections: 3000/137060 Jun 06 19:31:33 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:33 collections: 4000/137060 Jun 06 19:31:34 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:34 collections: 5000/137060 Jun 06 19:31:35 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:35 collections: 6000/137060 Jun 06 19:31:36 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:36 collections: 7000/137060 Jun 06 19:31:36 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:36 collections: 8000/137060 Jun 06 19:31:37 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:37 collections: 9000/137060 Jun 06 19:31:38 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:38 collections: 10000/137060 Jun 06 19:31:39 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:39 collections: 11000/137060 Jun 06 19:31:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:40 collections: 12000/137060 Jun 06 19:31:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:40 collections: 13000/137060 Jun 06 19:31:41 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:41 collections: 14000/137060 Jun 06 19:31:42 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:42 collections: 15000/137060 Jun 06 19:31:43 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:43 collections: 16000/137060 Jun 06 19:31:44 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:44 collections: 17000/137060 Jun 06 19:31:45 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:45 collections: 18000/137060 Jun 06 19:31:45 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:45 collections: 19000/137060 Jun 06 19:31:47 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:47 collections: 20000/137060 Jun 06 19:31:49 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:49 collections: 21000/137060 Jun 06 19:31:51 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:51 collections: 22000/137060 Jun 06 19:31:52 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:52 collections: 23000/137060 Jun 06 19:31:53 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:53 collections: 24000/137060 Jun 06 19:31:54 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:54 collections: 25000/137060 Jun 06 19:31:55 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:55 collections: 26000/137060 Jun 06 19:31:56 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:56 collections: 27000/137060 Jun 06 19:31:57 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:57 collections: 28000/137060 Jun 06 19:31:58 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:58 collections: 29000/137060 Jun 06 19:31:58 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:58 collections: 30000/137060 Jun 06 19:31:59 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:31:59 collections: 31000/137060 Jun 06 19:32:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:00 collections: 32000/137060 Jun 06 19:32:01 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:01 collections: 33000/137060 Jun 06 19:32:02 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:02 collections: 34000/137060 Jun 06 19:32:03 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:03 collections: 35000/137060 Jun 06 19:32:04 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:04 collections: 36000/137060 Jun 06 19:32:05 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:05 collections: 37000/137060 Jun 06 19:32:06 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:06 collections: 38000/137060 Jun 06 19:32:07 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:07 collections: 39000/137060 Jun 06 19:32:08 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:08 collections: 40000/137060 Jun 06 19:32:09 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:09 collections: 41000/137060 Jun 06 19:32:10 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:10 collections: 42000/137060 Jun 06 19:32:11 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:11 collections: 43000/137060 Jun 06 19:32:12 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:12 collections: 44000/137060 Jun 06 19:32:13 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:13 collections: 45000/137060 Jun 06 19:32:14 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:14 collections: 46000/137060 Jun 06 19:32:15 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:15 collections: 47000/137060 Jun 06 19:32:15 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:15 collections: 48000/137060 Jun 06 19:32:16 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:16 collections: 49000/137060 Jun 06 19:32:17 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:17 collections: 50000/137060 Jun 06 19:32:18 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:18 collections: 51000/137060 Jun 06 19:32:19 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:19 collections: 52000/137060 Jun 06 19:32:20 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:20 collections: 53000/137060 Jun 06 19:32:21 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:21 collections: 54000/137060 Jun 06 19:32:22 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:22 collections: 55000/137060 Jun 06 19:32:23 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:23 collections: 56000/137060 Jun 06 19:32:24 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:24 collections: 57000/137060 Jun 06 19:32:25 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:25 collections: 58000/137060 Jun 06 19:32:26 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:26 collections: 59000/137060 Jun 06 19:32:27 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:27 collections: 60000/137060 Jun 06 19:32:27 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:27 collections: 61000/137060 Jun 06 19:32:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:28 collections: 62000/137060 Jun 06 19:32:29 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:29 collections: 63000/137060 Jun 06 19:32:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:30 collections: 64000/137060 Jun 06 19:32:31 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:31 collections: 65000/137060 Jun 06 19:32:32 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:32 collections: 66000/137060 Jun 06 19:32:33 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:33 collections: 67000/137060 Jun 06 19:32:34 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:34 collections: 68000/137060 Jun 06 19:32:35 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:35 collections: 69000/137060 Jun 06 19:32:36 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:36 collections: 70000/137060 Jun 06 19:32:37 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:37 collections: 71000/137060 Jun 06 19:32:38 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:38 collections: 72000/137060 Jun 06 19:32:39 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:39 collections: 73000/137060 Jun 06 19:32:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:40 collections: 74000/137060 Jun 06 19:32:41 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:41 collections: 75000/137060 Jun 06 19:32:42 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:42 collections: 76000/137060 Jun 06 19:32:43 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:43 collections: 77000/137060 Jun 06 19:32:44 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:44 collections: 78000/137060 Jun 06 19:32:45 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:45 collections: 79000/137060 Jun 06 19:32:46 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:46 collections: 80000/137060 Jun 06 19:32:46 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:46 collections: 81000/137060 Jun 06 19:32:48 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:48 collections: 82000/137060 Jun 06 19:32:48 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:48 collections: 83000/137060 Jun 06 19:32:49 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:49 collections: 84000/137060 Jun 06 19:32:50 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:50 collections: 85000/137060 Jun 06 19:32:51 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:51 collections: 86000/137060 Jun 06 19:32:51 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:51 collections: 87000/137060 Jun 06 19:32:52 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:52 collections: 88000/137060 Jun 06 19:32:53 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:53 collections: 89000/137060 Jun 06 19:32:54 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:54 collections: 90000/137060 Jun 06 19:32:55 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:55 collections: 91000/137060 Jun 06 19:32:56 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:56 collections: 92000/137060 Jun 06 19:32:57 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:57 collections: 93000/137060 Jun 06 19:32:57 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:57 collections: 94000/137060 Jun 06 19:32:58 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:58 collections: 95000/137060 Jun 06 19:32:59 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:59 collections: 96000/137060 Jun 06 19:32:59 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:32:59 collections: 97000/137060 Jun 06 19:33:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:00 collections: 98000/137060 Jun 06 19:33:01 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:01 collections: 99000/137060 Jun 06 19:33:02 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:02 collections: 100000/137060 Jun 06 19:33:03 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:03 collections: 101000/137060 Jun 06 19:33:03 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:03 collections: 102000/137060 Jun 06 19:33:04 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:04 collections: 103000/137060 Jun 06 19:33:05 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:05 collections: 104000/137060 Jun 06 19:33:06 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:06 collections: 105000/137060 Jun 06 19:33:06 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:06 collections: 106000/137060 Jun 06 19:33:07 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:07 collections: 107000/137060 Jun 06 19:33:08 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:08 collections: 108000/137060 Jun 06 19:33:09 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:09 collections: 109000/137060 Jun 06 19:33:09 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:09 collections: 110000/137060 Jun 06 19:33:10 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:10 collections: 111000/137060 Jun 06 19:33:11 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:11 collections: 112000/137060 Jun 06 19:33:12 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:12 collections: 113000/137060 Jun 06 19:33:12 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:12 collections: 114000/137060 Jun 06 19:33:13 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:13 collections: 115000/137060 Jun 06 19:33:14 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:14 collections: 116000/137060 Jun 06 19:33:14 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:14 collections: 117000/137060 Jun 06 19:33:15 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:15 collections: 118000/137060 Jun 06 19:33:16 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:16 collections: 119000/137060 Jun 06 19:33:16 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:16 collections: 120000/137060 Jun 06 19:33:17 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:17 collections: 121000/137060 Jun 06 19:33:18 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:18 collections: 122000/137060 Jun 06 19:33:19 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:19 collections: 123000/137060 Jun 06 19:33:19 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:19 collections: 124000/137060 Jun 06 19:33:20 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:20 collections: 125000/137060 Jun 06 19:33:21 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:21 collections: 126000/137060 Jun 06 19:33:21 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:21 collections: 127000/137060 Jun 06 19:33:22 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:22 collections: 128000/137060 Jun 06 19:33:23 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:23 collections: 129000/137060 Jun 06 19:33:24 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:24 collections: 130000/137060 Jun 06 19:33:24 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:24 collections: 131000/137060 Jun 06 19:33:25 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:25 collections: 132000/137060 Jun 06 19:33:26 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:26 collections: 133000/137060 Jun 06 19:33:26 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:26 collections: 134000/137060 Jun 06 19:33:27 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:27 collections: 135000/137060 Jun 06 19:33:29 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:29 collections: 136000/137060 Jun 06 19:33:31 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:31 collections: 137000/137060 Jun 06 19:33:31 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:31 collections: 137060/137060 Jun 06 19:33:31 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:33:31 collections: 137060/137060 Jun 06 19:37:16 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:37:16 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 19:37:17 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:37:17 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:37:17 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:37:17 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:39:48 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:39:48 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 19:39:48 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:39:48 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:39:48 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:39:48 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:39:54 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:39:54 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 19:39:54 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:39:54 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:39:54 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:39:54 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:41:17 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:41:17 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 19:41:17 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:41:17 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:41:17 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:41:17 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:46:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:46:40 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 19:46:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:46:40 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:46:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:46:40 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:47:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:47:00 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 19:47:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:47:00 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:47:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:47:00 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:47:55 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:47:55 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 19:47:55 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:47:55 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:47:55 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:47:55 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:48:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:48:00 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 19:48:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:48:00 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:48:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:48:00 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:54:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:54:40 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 19:54:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:54:40 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:54:40 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:54:40 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:54:55 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:54:55 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 06 19:54:55 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:54:55 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:54:55 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:54:55 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:57:24 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:57:24 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 19:57:24 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:57:24 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:57:24 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:57:24 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 19:57:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:57:30 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 06 19:57:31 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:57:31 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 19:57:31 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 19:57:31 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): retrieve index Jun 06 20:02:32 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:32 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 20:02:32 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:32 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 20:02:32 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:32 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): done Jun 06 20:02:56 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:56 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 06 20:02:56 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:56 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 20:02:56 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:56 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): done Jun 06 20:02:58 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:58 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 20:02:58 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:58 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 20:02:58 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:02:58 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): done Jun 06 20:03:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:00 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 06 20:03:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:00 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 20:03:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:00 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): done Jun 06 20:03:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:00 GetCurrentState: took 31m30.280896175s Jun 06 20:03:00 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:00 ComputeChangeSets: start Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 ComputeChangeSets: took 28.336841186s Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 === Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 0 replicas (0 blocks, 0 bytes) lost (0=have<want) Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 3357390 replicas (3357390 blocks, 199210561559859 bytes) underreplicated (0<have<want) Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 292 replicas (146 blocks, 2035303304 bytes) just right (have=want) Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 4 replicas (4 blocks, 318710 bytes) overreplicated (have>want>0) Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new) Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old) Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 === Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 storage class "default": 6715080 replicas (3357540 blocks, 398423158486764 bytes) desired Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 storage class "default": 3357390 replicas (3357390 blocks, 199210561559859 bytes) short Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 storage class "default": 4 replicas (4 blocks, 31871 bytes) surplus Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 storage class "default": 0 replicas (0 blocks, 0 bytes) unachievable Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 === Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 6715080 replicas (3357540 blocks, 398423158486764 bytes) total commitment (excluding unreferenced) Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 13430776 replicas (3357540 blocks, 796850387835104 bytes) total usage Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 === Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:839114, Trashes:0} Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:840514, Trashes:0} Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:839577, Trashes:0} Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:838185, Trashes:0} Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 === Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 Replication level distribution (counting N replicas on a single server as N): Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 0: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 1: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 2: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 3: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 4: 3357390 ########################################################### Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 5: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 6: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 7: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 8: 146 ################### Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 9: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 10: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 11: 0 Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 12: 4 ###### Jun 06 20:03:28 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:28 === Jun 06 20:03:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:30 Run: took 32m0.247939103s Jun 06 20:03:30 dhhck.arvadosapi.com keep-balance[25011]: 2018/06/06 20:03:30 run succeeded
Updated by Ward Vandewege over 6 years ago
Testing new da40bd0960806df8e2799e4fb716d41ad08b169f
keep-balance from this branch, with duplicate mounts on the keepstores (all mounts rw on all keepstores):
Jun 07 13:08:00 dhhck.arvadosapi.com systemd[1]: Started Arvados Keep Balance. Jun 07 13:08:00 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:00 keep-balance dev started Jun 07 13:08:00 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:00 starting up: will scan every 4h0m0s and on SIGUSR1 Jun 07 13:08:00 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:00 WARNING: Will scan periodically, but no changes will be committed. Jun 07 13:08:00 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:00 ======= Consider using -commit-pulls and -commit-trash flags. Jun 07 13:08:00 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:00 Run: start Jun 07 13:08:01 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:01 skipping dhhck-bi6l4-blb6dtx0sx07d7k with service type "proxy" Jun 07 13:08:01 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:01 GetCurrentState: start Jun 07 13:08:01 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:01 mount zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3) Jun 07 13:08:01 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:01 mount zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3) Jun 07 13:08:01 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:01 mount zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3) Jun 07 13:08:01 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:01 mount zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): retrieve index from dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3) Jun 07 13:08:02 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:02 collections: 0/137060 Jun 07 13:08:04 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:04 collections: 1000/137060 Jun 07 13:08:05 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:05 collections: 2000/137060 Jun 07 13:08:06 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:06 collections: 3000/137060 Jun 07 13:08:07 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:07 collections: 4000/137060 Jun 07 13:08:08 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:08 collections: 5000/137060 Jun 07 13:08:09 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:09 collections: 6000/137060 Jun 07 13:08:09 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:09 collections: 7000/137060 Jun 07 13:08:10 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:10 collections: 8000/137060 Jun 07 13:08:11 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:11 collections: 9000/137060 Jun 07 13:08:12 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:12 collections: 10000/137060 Jun 07 13:08:13 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:13 collections: 11000/137060 Jun 07 13:08:13 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:13 collections: 12000/137060 Jun 07 13:08:14 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:14 collections: 13000/137060 Jun 07 13:08:15 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:15 collections: 14000/137060 Jun 07 13:08:16 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:16 collections: 15000/137060 Jun 07 13:08:17 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:17 collections: 16000/137060 Jun 07 13:08:17 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:17 collections: 17000/137060 Jun 07 13:08:18 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:18 collections: 18000/137060 Jun 07 13:08:19 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:19 collections: 19000/137060 Jun 07 13:08:21 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:21 collections: 20000/137060 Jun 07 13:08:23 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:23 collections: 21000/137060 Jun 07 13:08:25 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:25 collections: 22000/137060 Jun 07 13:08:26 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:26 collections: 23000/137060 Jun 07 13:08:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:27 collections: 24000/137060 Jun 07 13:08:28 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:28 collections: 25000/137060 Jun 07 13:08:29 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:29 collections: 26000/137060 Jun 07 13:08:30 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:30 collections: 27000/137060 Jun 07 13:08:31 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:31 collections: 28000/137060 Jun 07 13:08:32 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:32 collections: 29000/137060 Jun 07 13:08:33 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:33 collections: 30000/137060 Jun 07 13:08:33 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:33 collections: 31000/137060 Jun 07 13:08:34 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:34 collections: 32000/137060 Jun 07 13:08:35 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:35 collections: 33000/137060 Jun 07 13:08:36 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:36 collections: 34000/137060 Jun 07 13:08:37 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:37 collections: 35000/137060 Jun 07 13:08:38 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:38 collections: 36000/137060 Jun 07 13:08:39 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:39 collections: 37000/137060 Jun 07 13:08:40 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:40 collections: 38000/137060 Jun 07 13:08:41 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:41 collections: 39000/137060 Jun 07 13:08:42 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:42 collections: 40000/137060 Jun 07 13:08:43 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:43 collections: 41000/137060 Jun 07 13:08:44 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:44 collections: 42000/137060 Jun 07 13:08:45 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:45 collections: 43000/137060 Jun 07 13:08:46 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:46 collections: 44000/137060 Jun 07 13:08:47 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:47 collections: 45000/137060 Jun 07 13:08:48 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:48 collections: 46000/137060 Jun 07 13:08:49 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:49 collections: 47000/137060 Jun 07 13:08:50 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:50 collections: 48000/137060 Jun 07 13:08:51 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:51 collections: 49000/137060 Jun 07 13:08:52 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:52 collections: 50000/137060 Jun 07 13:08:52 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:52 collections: 51000/137060 Jun 07 13:08:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:53 collections: 52000/137060 Jun 07 13:08:54 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:54 collections: 53000/137060 Jun 07 13:08:55 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:55 collections: 54000/137060 Jun 07 13:08:56 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:56 collections: 55000/137060 Jun 07 13:08:57 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:57 collections: 56000/137060 Jun 07 13:08:58 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:58 collections: 57000/137060 Jun 07 13:08:59 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:08:59 collections: 58000/137060 Jun 07 13:09:00 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:00 collections: 59000/137060 Jun 07 13:09:01 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:01 collections: 60000/137060 Jun 07 13:09:02 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:02 collections: 61000/137060 Jun 07 13:09:03 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:03 collections: 62000/137060 Jun 07 13:09:04 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:04 collections: 63000/137060 Jun 07 13:09:05 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:05 collections: 64000/137060 Jun 07 13:09:05 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:05 collections: 65000/137060 Jun 07 13:09:06 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:06 collections: 66000/137060 Jun 07 13:09:07 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:07 collections: 67000/137060 Jun 07 13:09:08 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:08 collections: 68000/137060 Jun 07 13:09:09 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:09 collections: 69000/137060 Jun 07 13:09:10 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:10 collections: 70000/137060 Jun 07 13:09:11 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:11 collections: 71000/137060 Jun 07 13:09:12 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:12 collections: 72000/137060 Jun 07 13:09:13 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:13 collections: 73000/137060 Jun 07 13:09:14 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:14 collections: 74000/137060 Jun 07 13:09:15 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:15 collections: 75000/137060 Jun 07 13:09:16 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:16 collections: 76000/137060 Jun 07 13:09:17 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:17 collections: 77000/137060 Jun 07 13:09:18 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:18 collections: 78000/137060 Jun 07 13:09:19 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:19 collections: 79000/137060 Jun 07 13:09:20 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:20 collections: 80000/137060 Jun 07 13:09:21 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:21 collections: 81000/137060 Jun 07 13:09:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:22 collections: 82000/137060 Jun 07 13:09:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:22 collections: 83000/137060 Jun 07 13:09:23 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:23 collections: 84000/137060 Jun 07 13:09:24 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:24 collections: 85000/137060 Jun 07 13:09:25 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:25 collections: 86000/137060 Jun 07 13:09:26 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:26 collections: 87000/137060 Jun 07 13:09:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:27 collections: 88000/137060 Jun 07 13:09:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:27 collections: 89000/137060 Jun 07 13:09:28 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:28 collections: 90000/137060 Jun 07 13:09:30 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:30 collections: 91000/137060 Jun 07 13:09:31 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:31 collections: 92000/137060 Jun 07 13:09:31 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:31 collections: 93000/137060 Jun 07 13:09:32 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:32 collections: 94000/137060 Jun 07 13:09:33 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:33 collections: 95000/137060 Jun 07 13:09:33 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:33 collections: 96000/137060 Jun 07 13:09:34 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:34 collections: 97000/137060 Jun 07 13:09:35 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:35 collections: 98000/137060 Jun 07 13:09:36 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:36 collections: 99000/137060 Jun 07 13:09:37 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:37 collections: 100000/137060 Jun 07 13:09:37 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:37 collections: 101000/137060 Jun 07 13:09:38 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:38 collections: 102000/137060 Jun 07 13:09:38 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:38 collections: 103000/137060 Jun 07 13:09:39 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:39 collections: 104000/137060 Jun 07 13:09:40 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:40 collections: 105000/137060 Jun 07 13:09:41 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:41 collections: 106000/137060 Jun 07 13:09:41 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:41 collections: 107000/137060 Jun 07 13:09:42 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:42 collections: 108000/137060 Jun 07 13:09:43 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:43 collections: 109000/137060 Jun 07 13:09:43 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:43 collections: 110000/137060 Jun 07 13:09:45 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:45 collections: 112000/137060 Jun 07 13:09:45 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:45 collections: 113000/137060 Jun 07 13:09:46 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:46 collections: 114000/137060 Jun 07 13:09:47 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:47 collections: 115000/137060 Jun 07 13:09:48 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:48 collections: 116000/137060 Jun 07 13:09:48 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:48 collections: 117000/137060 Jun 07 13:09:49 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:49 collections: 118000/137060 Jun 07 13:09:50 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:50 collections: 119000/137060 Jun 07 13:09:51 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:51 collections: 120000/137060 Jun 07 13:09:51 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:51 collections: 121000/137060 Jun 07 13:09:52 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:52 collections: 122000/137060 Jun 07 13:09:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:53 collections: 123000/137060 Jun 07 13:09:54 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:54 collections: 124000/137060 Jun 07 13:09:54 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:54 collections: 125000/137060 Jun 07 13:09:55 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:55 collections: 126000/137060 Jun 07 13:09:56 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:56 collections: 127000/137060 Jun 07 13:09:56 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:56 collections: 128000/137060 Jun 07 13:09:57 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:57 collections: 129000/137060 Jun 07 13:09:58 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:58 collections: 130000/137060 Jun 07 13:09:59 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:59 collections: 131000/137060 Jun 07 13:09:59 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:09:59 collections: 132000/137060 Jun 07 13:10:00 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:10:00 collections: 133000/137060 Jun 07 13:10:00 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:10:00 collections: 134000/137060 Jun 07 13:10:01 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:10:01 collections: 135000/137060 Jun 07 13:10:03 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:10:03 collections: 136000/137060 Jun 07 13:10:06 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:10:06 collections: 137000/137060 Jun 07 13:10:06 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:10:06 collections: 137060/137060 Jun 07 13:10:06 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:10:06 collections: 137060/137060 Jun 07 13:14:38 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:38 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 07 13:14:38 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:38 zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 512782 replicas Jun 07 13:14:38 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:38 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 07 13:14:38 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:38 zzzzz-ivpuk-j0wrroy6q6cwkl0 (s3:///dhhck-keep-1) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 512782 replicas Jun 07 13:14:38 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:38 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 07 13:14:39 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:39 zzzzz-ivpuk-f9uopk2c45y8epe (s3:///dhhck-keep-1) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 512782 replicas Jun 07 13:14:39 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:39 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 512782 replicas to map Jun 07 13:14:39 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:39 zzzzz-ivpuk-jtb935k4e3gb5ki (s3:///dhhck-keep-1) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 512782 replicas Jun 07 13:14:39 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:14:39 mount zzzzz-ivpuk-1yi1txjx0knwcmp (s3:///dhhck-keep-1) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): index done Jun 07 13:17:26 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:26 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 07 13:17:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:27 zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 898276 replicas Jun 07 13:17:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:27 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 07 13:17:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:27 zzzzz-ivpuk-mngd73gwg90o0cc (s3:///dhhck-keep-2) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 898276 replicas Jun 07 13:17:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:27 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 07 13:17:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:27 zzzzz-ivpuk-qkmuclkk0h82cew (s3:///dhhck-keep-2) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 898276 replicas Jun 07 13:17:27 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:27 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 898276 replicas to map Jun 07 13:17:28 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:28 zzzzz-ivpuk-c06sk8mabam179f (s3:///dhhck-keep-2) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 898276 replicas Jun 07 13:17:28 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:28 mount zzzzz-ivpuk-cy9ao3ko41dw91v (s3:///dhhck-keep-2) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): index done Jun 07 13:17:32 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:32 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 07 13:17:32 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:32 zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 902720 replicas Jun 07 13:17:32 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:32 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 07 13:17:32 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:32 zzzzz-ivpuk-ob1g5c4cpgdsesb (s3:///dhhck-keep-3) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 902720 replicas Jun 07 13:17:32 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:32 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 07 13:17:33 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:33 zzzzz-ivpuk-y171idzj2qzfhx6 (s3:///dhhck-keep-3) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 902720 replicas Jun 07 13:17:33 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:33 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 902720 replicas to map Jun 07 13:17:33 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:33 zzzzz-ivpuk-vrz7p9czi0i31mu (s3:///dhhck-keep-3) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 902720 replicas Jun 07 13:17:33 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:17:33 mount zzzzz-ivpuk-mp6rjct1zptgr7m (s3:///dhhck-keep-3) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): index done Jun 07 13:18:52 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:52 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 07 13:18:52 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:52 zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas Jun 07 13:18:52 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:52 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 07 13:18:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:53 zzzzz-ivpuk-yjkc573a4bzdhav (s3:///dhhck-keep-0) on dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas Jun 07 13:18:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:53 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 07 13:18:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:53 zzzzz-ivpuk-5lcgxnqbpbnesm4 (s3:///dhhck-keep-0) on dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas Jun 07 13:18:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:53 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): add 1043916 replicas to map Jun 07 13:18:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:53 zzzzz-ivpuk-z4ar6aoj1y3qsr5 (s3:///dhhck-keep-0) on dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): added 1043916 replicas Jun 07 13:18:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:53 mount zzzzz-ivpuk-nb5izg026aazilj (s3:///dhhck-keep-0) on dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): index done Jun 07 13:18:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:53 GetCurrentState: took 10m52.5263035s Jun 07 13:18:53 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:18:53 ComputeChangeSets: start Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 ComputeChangeSets: took 28.242432124s Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 === Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 0 replicas (0 blocks, 0 bytes) lost (0=have<want) Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 3357390 replicas (3357390 blocks, 199210561559859 bytes) underreplicated (0<have<want) Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 292 replicas (146 blocks, 2035303304 bytes) just right (have=want) Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 4 replicas (4 blocks, 318710 bytes) overreplicated (have>want>0) Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 0 replicas (0 blocks, 0 bytes) unreferenced (have>want=0, new) Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 0 replicas (0 blocks, 0 bytes) garbage (have>want=0, old) Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 === Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 storage class "default": 6715080 replicas (3357540 blocks, 398423158486764 bytes) desired Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 storage class "default": 3357390 replicas (3357390 blocks, 199210561559859 bytes) short Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 storage class "default": 4 replicas (4 blocks, 31871 bytes) surplus Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 storage class "default": 0 replicas (0 blocks, 0 bytes) unachievable Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 === Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 6715080 replicas (3357540 blocks, 398423158486764 bytes) total commitment (excluding unreferenced) Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 13430776 replicas (3357540 blocks, 796850387835104 bytes) total usage Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 === Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 dhhck-bi6l4-oynapdlh4hzydcf (keep0.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:839577, Trashes:0} Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 dhhck-bi6l4-jt6xs3d6d531oaa (keep1.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:838185, Trashes:0} Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 dhhck-bi6l4-pkwwh8mhe0qgmu6 (keep2.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:839114, Trashes:0} Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 dhhck-bi6l4-u6ez2qmdvdde1xc (keep3.dhhck.arvadosapi.com:25107, s3): ChangeSet{Pulls:840514, Trashes:0} Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 === Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 Replication level distribution (counting N replicas on a single server as N): Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 0: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 1: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 2: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 3: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 4: 3357390 ########################################################### Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 5: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 6: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 7: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 8: 146 ################### Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 9: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 10: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 11: 0 Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 12: 4 ###### Jun 07 13:19:22 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:22 === Jun 07 13:19:23 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:23 Run: took 11m22.814890766s Jun 07 13:19:23 dhhck.arvadosapi.com keep-balance[29484]: 2018/06/07 13:19:23 run succeeded
Updated by Tom Clegg over 6 years ago
- Status changed from In Progress to Resolved
- Remaining (hours) set to 0.0
Actions