Arvados

Found the root cause of the error described above (a malformed crunch-dispatch-local-credentials file, fixed in arvados-formula:commit:ba3827)

Refactored the provisioning script and configuration files:

• pillars: SSL certs and keys are included now on each pillar, and not as a snippet, so we can iterate over them in the provision script
• split keepweb nginx's configuration in its two parts (download and collections) for the same reason.
• added a new state (custom_certs) which copies the certs from a dir where the user uploads them.
• refactored the snakeoil_certs in /single_host/multiple_hostnames to generate individual certs that can be "uploaded" by custom_certs, so we can test it.
• added a --development parameter to the the provision.sh script, to include the snakeoil certs when testing.
• added documentation.

15a2556dc (branch 17742-provide-custom-certs)

Reviewing f54cc984969657be50c093b917feb49a19d78c22

In doc/install/salt-multi-host.html.textile.liquid

+The <i>multi_host</i> include LetsEncrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.

There seems to be a word missing before "include".

+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>downoad / collections</i>):

Typo: "download".

doc/install/salt-single-host.html.textile.liquid

Does this method not include Let's Encrypt salt code? It doesn't seem mentioned in that file.

+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>downoad / collections</i>):

Typo: "download"

In tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls

       # required to test with arvados-snakeoil certs
-      insecure: true
+      insecure: false

Since you're changing it, is that comment still relevant? If so what does "test" mean? Automated tests? A user trying out Arvados? Something else? Is this what you mean: "When using arvados-snakeoil certs set insecure: true" ?

In tools/salt-install/provision.sh

+          grep -q ${R} ${P_DIR}/extra_custom_certs.sls || echo "  - ${R}" >> ${P_DIR}/extra_custom_certs.sls
+    
+          # As the pillar differ whether we use LE or custom certs, we need to do a final edition on them
+          # Special case for keepweb

That empty line has trailing whitespace...

I haven't tried it yet.

Is the plan to test the custom cert functionality with the automated test-deploy jobs on Jenkins?

Ward Vandewege wrote:

Reviewing f54cc984969657be50c093b917feb49a19d78c22

Addressed your suggestions on d68c3776f (branch 17742-provide-custom-certs)

I haven't tried it yet.

Is the plan to test the custom cert functionality with the automated test-deploy jobs on Jenkins?

Yes, the way I modified the code, the snake oil certificates are deployed using the custom-certs functionality:

The snakeoil_certs state file now creates certificates that are deployed in the destination directory where the custom_certs state file expect them to be (and where the docs specify they should be copied). From there, the latter will deploy them.

Thanks, a few more things:

• Please fix the spelling of Let's Encrypt, you have "LetsEncrypt" in three places in the docs, and that is wrong.

• The blurb about custom certificates in `doc/install/salt-multi-host.html.textile.liquid` and `doc/install/salt-single-host.html.textile.liquid` appears to be identical, please pull that out into an 'include', see the _includes directory for an example.

With those changes, LGTM thanks.