Project

General

Profile

Actions

Feature #17742

closed

[deployment][provision] allow to provide custom SSL certificates

Added by Javier Bértoli over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Deployment
Target version:
Start date:
10/13/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

The current provision script allow to get LE certificates but does not manage custom certificates.

The user should be able to add custom certificates and keys in a directory, they should be copied where appropriate and the nginx pillars modified to match this.


Files

salida_cluster_test_insecure_false (12.8 KB) salida_cluster_test_insecure_false test with insecure set to false Javier Bértoli, 10/08/2021 10:18 AM
salida_cluster_test_insecure_true (12.8 KB) salida_cluster_test_insecure_true test with insecure set to true Javier Bértoli, 10/08/2021 10:18 AM
arvados_config.yml (3.24 KB) arvados_config.yml Javier Bértoli, 10/08/2021 10:33 AM

Subtasks 1 (0 open1 closed)

Task #17873: Review commit d68c3776f (branch 17742-provide-custom-certs)ResolvedJavier Bértoli10/13/2021

Actions
Actions #1

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-06-09 sprint to 2021-06-23 sprint
Actions #2

Updated by Javier Bértoli over 3 years ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-06-23 sprint to 2021-07-07 sprint
Actions #4

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-07-07 sprint to 2021-07-21 sprint
Actions #5

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-07-21 sprint to 2021-08-04 sprint
Actions #6

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-08-04 sprint to 2021-08-18 sprint
Actions #7

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-08-18 sprint to 2021-09-01 sprint
Actions #8

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-09-01 sprint to 2021-09-15 sprint
Actions #9

Updated by Peter Amstutz over 3 years ago

  • Target version changed from 2021-09-15 sprint to 2021-09-29 sprint
Actions #10

Updated by Peter Amstutz about 3 years ago

  • Release set to 42
Actions #11

Updated by Peter Amstutz about 3 years ago

  • Target version changed from 2021-09-29 sprint to 2021-10-13 sprint

Updated by Javier Bértoli about 3 years ago

I'm experiencing some errors when running the final cluster_tests (attached)

Actions #14

Updated by Javier Bértoli about 3 years ago

Found the root cause of the error described above (a malformed crunch-dispatch-local-credentials file, fixed in arvados-formula:commit:ba3827)

Refactored the provisioning script and configuration files:

  • pillars: SSL certs and keys are included now on each pillar, and not as a snippet, so we can iterate over them in the provision script
  • split keepweb nginx's configuration in its two parts (download and collections) for the same reason.
  • added a new state (custom_certs) which copies the certs from a dir where the user uploads them.
  • refactored the snakeoil_certs in /single_host/multiple_hostnames to generate individual certs that can be "uploaded" by custom_certs, so we can test it.
  • added a --development parameter to the the provision.sh script, to include the snakeoil certs when testing.
  • added documentation.

15a2556dc (branch 17742-provide-custom-certs)

Actions #15

Updated by Peter Amstutz about 3 years ago

  • Target version changed from 2021-10-13 sprint to 2021-10-27 sprint
Actions #16

Updated by Ward Vandewege about 3 years ago

Reviewing f54cc984969657be50c093b917feb49a19d78c22

In doc/install/salt-multi-host.html.textile.liquid

+The <i>multi_host</i> include LetsEncrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.

There seems to be a word missing before "include".

+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>downoad / collections</i>):

Typo: "download".

doc/install/salt-single-host.html.textile.liquid

Does this method not include Let's Encrypt salt code? It doesn't seem mentioned in that file.

+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>downoad / collections</i>):

Typo: "download"

In tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls

       # required to test with arvados-snakeoil certs
-      insecure: true
+      insecure: false

Since you're changing it, is that comment still relevant? If so what does "test" mean? Automated tests? A user trying out Arvados? Something else? Is this what you mean: "When using arvados-snakeoil certs set insecure: true" ?

In tools/salt-install/provision.sh

+          grep -q ${R} ${P_DIR}/extra_custom_certs.sls || echo "  - ${R}" >> ${P_DIR}/extra_custom_certs.sls
+    
+          # As the pillar differ whether we use LE or custom certs, we need to do a final edition on them
+          # Special case for keepweb

That empty line has trailing whitespace...

I haven't tried it yet.

Is the plan to test the custom cert functionality with the automated test-deploy jobs on Jenkins?

Actions #17

Updated by Javier Bértoli about 3 years ago

Ward Vandewege wrote:

Reviewing f54cc984969657be50c093b917feb49a19d78c22

Addressed your suggestions on d68c3776f (branch 17742-provide-custom-certs)

I haven't tried it yet.

Is the plan to test the custom cert functionality with the automated test-deploy jobs on Jenkins?

Yes, the way I modified the code, the snake oil certificates are deployed using the custom-certs functionality:

The snakeoil_certs state file now creates certificates that are deployed in the destination directory where the custom_certs state file expect them to be (and where the docs specify they should be copied). From there, the latter will deploy them.

Actions #18

Updated by Ward Vandewege about 3 years ago

Thanks, a few more things:

  • Please fix the spelling of Let's Encrypt, you have "LetsEncrypt" in three places in the docs, and that is wrong.
  • The blurb about custom certificates in `doc/install/salt-multi-host.html.textile.liquid` and `doc/install/salt-single-host.html.textile.liquid` appears to be identical, please pull that out into an 'include', see the _includes directory for an example.

With those changes, LGTM thanks.

Actions #19

Updated by Javier Bértoli about 3 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100
Actions #20

Updated by Peter Amstutz about 3 years ago

  • Release changed from 42 to 45
Actions

Also available in: Atom PDF