Bug #7254
Updated by Brett Smith over 9 years ago
Doing a simple arv-put that specifies --replication different than the default will ignore the replication for Keep blocks. They'll still be replicated to Keep at the default level. Here's a A demonstration on the PGP tutorial data. The debug output shows it writing each block to two Keep services (each of two ports on keep0): data: <pre> brett@shell.4xphq:~/keep/by_id/c1bad4b39ca5a924e481008009d94e32+210$ ARVADOS_DEBUG=1 arv-put --replication 1 var-GS000016015-ASM.tsv.bz2 0M / 216M 0.0% 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: {u'4xphq-bi6l4-ospxuln4ial4svv': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospx uln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u 'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u' service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.c om'}, u'4xphq-bi6l4-dk9mjspdg2v8mhq': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvado sapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-t pzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service _ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}} 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: [{u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}, {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospxuln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}] 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: 204e43b8a1185621ca55a94839582e6f+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140344599521024)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/204e43b8a1185621ca55a94839582e6f 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-2, started 140344591128320)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 2015-09-10 13:49:05 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/204e43b8a1185621ca55a94839582e6f 2015-09-10 13:49:09 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140344599521024)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:49:09 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-2, started 140344591128320)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 64M / 216M 29.5% 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: b9677abbac956bd3e86b1deb28dfac03+67108864: ['http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4xphq.arvadosapi.com:25108/'] 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140344591128320)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/b9677abbac956bd3e86b1deb28dfac03 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-4, started 140344599521024)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 2015-09-10 13:49:10 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/b9677abbac956bd3e86b1deb28dfac03 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-4, started 140344599521024)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140344591128320)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 128M / 216M 59.1% 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: fc15aff2a762b13f521baf042140acec+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140344591128320)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/fc15aff2a762b13f521baf042140acec 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-6, started 140344599521024)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 2015-09-10 13:49:14 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/fc15aff2a762b13f521baf042140acec 2015-09-10 13:49:17 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-6, started 140344599521024)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25108/ 2015-09-10 13:49:17 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140344591128320)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 192M / 216M 88.6% 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: 323d2a3ce20370c4ca1d3462a344f8fd+25885655: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140344591128320)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/323d2a3ce20370c4ca1d3462a344f8fd 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-8, started 140344599521024)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25108/ 2015-09-10 13:49:18 arvados.keep[15725] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25108/323d2a3ce20370c4ca1d3462a344f8fd 2015-09-10 13:49:19 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-8, started 140344599521024)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25108/ 2015-09-10 13:49:19 arvados.keep[15725] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140344591128320)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/ 216M / 216M 100.0% Collection saved as 'Saved at 2015-09-10 13:49:04 UTC by brett@shell.4xphq.arvadosapi.com' 4xphq-4zz18-q0u4nb9z66joxs6 </pre> Same thing happens with more-than-default replication (e.g., --replication=3). @replication_desired@ is set correctly on the output collection, but Keep blocks are always replicated twice. It works if you add --no-resume: <pre> brett@shell.4xphq:~/keep/by_id/c1bad4b39ca5a924e481008009d94e32+210$ ARVADOS_DEBUG=1 arv-put --replication 1 --no-resume var-GS000016015-ASM.tsv.bz2 0M / 216M 0.0% 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: {u'4xphq-bi6l4-ospxuln4ial4svv': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-ospx uln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_at': u'2014-05-21T18:59:47.654755000Z', u 'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u' service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.c om'}, u'4xphq-bi6l4-dk9mjspdg2v8mhq': {u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4xphq.arvado sapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-t pzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service _ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}} 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: [{u'read_only': False, u'kind': u'arvados#keepService', u'uuid': u'4xphq-bi6l4-dk9mjspdg2v8mhq', '_service_root': 'http://keep0.4x phq.arvadosapi.com:25107/', u'modified_at': u'2014-05-21T18:59:47.620295000Z', u'created_at': u'2014-05-21T18:59:47.620495000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-dk9mjspdg2v8mhq', u'etag': u'64l3d8njg0f6enhj7hzx432e5', u'service_port': 25107, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u'service_host': u'keep0.4xphq.arvadosapi.com'}, {u'read_only': False, u'kind': u'arvados# keepService', u'uuid': u'4xphq-bi6l4-ospxuln4ial4svv', '_service_root': 'http://keep0.4xphq.arvadosapi.com:25108/', u'modified_at': u'2014-05-21T18:59:47.654627000Z', u'created_ at': u'2014-05-21T18:59:47.654755000Z', u'modified_by_client_uuid': None, u'owner_uuid': u'4xphq-tpzed-2ruv9ywhc7ozn9a', u'href': u'/keep_services/4xphq-bi6l4-ospxuln4ial4svv', u'etag': u'9yds3yc08bd97gxht3s95q6ua', u'service_port': 25108, u'service_type': u'disk', u'service_ssl_flag': False, u'modified_by_user_uuid': u'4xphq-tpzed-000000000000000', u' service_host': u'keep0.4xphq.arvadosapi.com'}] 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: 204e43b8a1185621ca55a94839582e6f+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/' ] 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140608440813312)> proceeding 204e43b8a1185621ca55a94839582e6f+67108864 http:/ /keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:52:49 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/204e43b8a1185621ca55a94839582e6f 2015-09-10 13:52:50 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-1, started 140608440813312)> succeeded 204e43b8a1185621ca55a94839582e6f+67108864 http:// keep0.4xphq.arvadosapi.com:25107/ 64M / 216M 29.5% 2015-09-10 13:52:51 arvados.keep[16939] DEBUG: b9677abbac956bd3e86b1deb28dfac03+67108864: ['http://keep0.4xphq.arvadosapi.com:25107/', 'http://keep0.4xphq.arvad osapi.com:25108/'] 2015-09-10 13:52:51 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140608440813312)> proceeding b9677abbac956bd3e86b1deb28dfac03+67108864 http:/ /keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:52:51 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/b9677abbac956bd3e86b1deb28dfac03 2015-09-10 13:52:52 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-3, started 140608440813312)> succeeded b9677abbac956bd3e86b1deb28dfac03+67108864 http:// keep0.4xphq.arvadosapi.com:25107/ 128M / 216M 59.1% 2015-09-10 13:52:53 arvados.keep[16939] DEBUG: fc15aff2a762b13f521baf042140acec+67108864: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arva dosapi.com:25107/'] 2015-09-10 13:52:53 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140608440813312)> proceeding fc15aff2a762b13f521baf042140acec+67108864 http:/ /keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:52:53 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/fc15aff2a762b13f521baf042140acec 2015-09-10 13:52:54 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-5, started 140608440813312)> succeeded fc15aff2a762b13f521baf042140acec+67108864 http://keep0.4xphq.arvadosapi.com:25107/ 192M / 216M 88.6% 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: 323d2a3ce20370c4ca1d3462a344f8fd+25885655: ['http://keep0.4xphq.arvadosapi.com:25108/', 'http://keep0.4xphq.arvadosapi.com:25107/'] 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140608440813312)> proceeding 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/ 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: Request: PUT http://keep0.4xphq.arvadosapi.com:25107/323d2a3ce20370c4ca1d3462a344f8fd 2015-09-10 13:52:55 arvados.keep[16939] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started 140608440813312)> succeeded 323d2a3ce20370c4ca1d3462a344f8fd+25885655 http://keep0.4xphq.arvadosapi.com:25107/ 216M / 216M 100.0% Collection saved as 'Saved at 2015-09-10 13:52:48 UTC by brett@shell.4xphq.arvadosapi.com' 4xphq-4zz18-lxei9i58nv4gvo7 </pre>