Arvados

Ready to review at ab8acfe

Some implementation notes:

• This code has not been tested against a live GCE account. We can test when we get GCE credentials that allow us to spin nodes up and down.
• GCE does not permit filtering instances on tags the way AWS does, so filtering has not been implemented. Clients will need to list all instances and do their own filtering.
• GCE does implement a 'tag' concept, but it is a flat keyword list rather than a key/value structure. When a node is synced, the Arvados hostname is updated on the node by setting a tag 'hostname-arvados.fqdn.name'.
  • The underlying GCE library supports a setMetadata call that should allow us to use metadata to track this in a cleaner way, but the libcloud API doesn't support this call yet.
• GCE node access is controlled with "service accounts" rather than security groups. These can be defined in gce.example.cfg by setting the service_account setting to a JSON string with the desired service account list.

For reviewing: the GCE drivers and tests were developed by ripping out everything in the EC2 drivers that doesn't look like GCE. Accordingly, I strongly recommend:

diff services/nodemanager/arvnodeman/computenode/driver/ec2.py services/nodemanager/arvnodeman/computenode/driver/gce.py
diff services/nodemanager/tests/test_computenode_driver_ec2.py services/nodemanager/tests/test_computenode_driver_gce.py
diff services/nodemanager/doc/ec2.example.cfg services/nodemanager/doc/gce.example.cfg