Arvados

crunch-job has this code to build its docker run command:

     # Dynamically configure the container to use the host system as its
      # DNS server.  Get the host's global addresses from the ip command,
      # and turn them into docker --dns options using gawk.
      $command .=
          q{$(ip -o address show scope global |
              gawk 'match($4, /^([0-9\.:]+)\//, x){print "--dns", x[1]}') };

We have this code because our own deployments run dnsmasq to resolve internal network names for the API and Keep servers; and when Docker detects that the host uses 127.0.0.1 for DNS, it starts the container using Google DNS in resolv.conf.

However, this requires compute nodes to run a DNS server, which seems like a deployment complication we don't need. docker daemon accepts one or more --dns flags to let you set default DNS servers for containers instead of Google DNS. On Debian, it's easy to set this in /etc/default/docker.io. I expect most distributions have a similar mechanism.

Let's get this configuration out of code and into deployment. Steps to complete this story:

• Remove the code above from crunch-job.
• In the compute node installation documentation, add a section to note that the Docker daemon uses Google DNS by default, and whatever DNS server it uses must be able to resolve the cluster's API and Keep servers. The default can be changed by using the Docker daemon's --dns flags. Link to the Docker daemon docs for more information. I don't think we can document the best way to do this for every distribution, but any suggestions or guidance you can add are worth bonus points.