Arvados

Currently workbench provides “create a new collection by combining selected collections” and “create a new collection by combining selected collection files” functionality. Workbench gets the required collection manifest_text(s), generages the combined manifest_text, invokes save new collection with the combined manifest_text, gets the newly saved collection from server, and finally displays to the user. However, this implementation is not scalable. When very large collections are combined or several collections from a very large collection are combined, the workbench combining collections operations fails. Workbench fails with timeout errors while exchanging these large collection manifest texts with the api server. There are a few bugs reported about this: #4943 and #5614.

A potential solution:

• Provide a “create by combining” api method that performs the steps currently being performed by the workbench. Thus,
  • This method will take the selections (list of files selected from a collection, or list of collections selected)
  • Generate the manifest_text by combining these selections (using the ruby sdk)
  • Save a new collection with the combined manifest text and the owner_uuid provided
  • Return the newly created collection to the client

• Hence, this solution requires the exchange of only one manifest text between the api server and workbench (that of the newly created collection), and hence offering much better performance.