Arvados

I added some logging to the client and it shows that loading from JSON takes just seconds, but the deep object copy takes almost half an hour and the JSON dump takes OVER AN HOUR.

Investigate:
- more compact storage structure like a trie or b-tree
- cPickle for save/restore
- non- JSON storage format since JSON is overkill

2017-01-19 03:24:49 arvados.arv_put⁵⁷¹: About to do setup state
2017-01-19 03:24:49 arvados.arv_put⁵⁷¹: Loading JSON cache file
2017-01-19 03:24:53 arvados.arv_put⁵⁷¹: Done loading cache file
2017-01-19 03:24:53 arvados.arv_put⁵⁷¹: Loading local collection from manifest text
2017-01-19 03:25:08 arvados.arv_put⁵⁷¹: Done loading local collection
2017-01-19 03:25:08 arvados.arv_put⁵⁷¹: done setup state
2017-01-19 03:25:53 arvados.arv_put⁵⁷¹: Removing local files which don't exist any more from collection
2017-01-19 03:26:08 arvados.arv_put⁵⁷¹: Entering _update
2017-01-19 03:26:19 arvados.arv_put⁵⁷¹: getting nonfinal local manifest text
2017-01-19 03:27:06 arvados.arv_put⁵⁷¹: Done getting local manifest text
2017-01-19 03:27:06 arvados.arv_put⁵⁷¹: acquired lock doing deep copy self._state - 280 bytes
2017-01-19 03:53:13 arvados.arv_put⁵⁷¹: done deep copy - 26 minutes
2017-01-19 03:53:17 arvados.arv_put⁵⁷¹: About to do json dump
2017-01-19 04:55:52 arvados.arv_put⁵⁷¹: Leaving _save_state - 63 minutes
282275M / 636959M 44.3%
2017-01-19 04:55:56 arvados.arv_put⁵⁷¹ WARNING: Leaving _update

Updates at: 9c9b520
Tests: https://ci.curoverse.com/job/developer-run-tests/151/

A couple of optimizations have improved times a lot.
On my tests with a ~12MB cache, state saving times went down from ~14 secs to ~0.8 secs by just avoiding the use of copy.deepcopy() and json.dump() and replacing them with json.dumps()
As for the resume start delay, using a set() instead of a list on the code reduced time spent on checking if every file on the local collection existed on the local disk before starting to upload the rest.

Is there any reason self._file_paths can't be a set()? It looks like the only thing that would need to be changed is to update self._file_paths.append(filename) to self._file_paths.add(filename).

Caching change looks good, might want to add a comment with what you learned about the performance of deepcopy/dump/dumps.

Peter Amstutz wrote:

Is there any reason self._file_paths can't be a set()? It looks like the only thing that would need to be changed is to update self._file_paths.append(filename) to self._file_paths.add(filename).

I read that sets weren't performant on iterations as list are, but didn't tested it myself. As you mentioned, the difference between both is some microseconds so it's updated as you requested.

Caching change looks good, might want to add a comment with what you learned about the performance of deepcopy/dump/dumps.

Done.

Test run: https://ci.curoverse.com/job/developer-run-tests/153/
Updates: e41dc06