Support #16873
closed
Jenkins job that uses "kitchen test" to test salt formulas against all supported distros
100%
Description
A new jenkins pipeline job which takes a given salt formula version and a given Arvados version (including dev) and performs installation and correctness checking using Dockerized environments of each supported distro.
Updated by Peter Amstutz about 4 years ago
- Tracker changed from Bug to Support
- Subject changed from Jenkins job that test-runs salt install in Docker against all supported distros to Jenkins job that uses "kitchen test" to test salt formulas against all supported distros
- Description updated (diff)
Updated by Peter Amstutz about 4 years ago
- Target version changed from 2020-10-07 Sprint to 2020-10-21 Sprint
Updated by Peter Amstutz about 4 years ago
- Target version changed from 2020-10-21 Sprint to 2020-11-04 Sprint
Updated by Peter Amstutz about 4 years ago
- Target version changed from 2020-11-04 Sprint to 2020-11-18
Updated by Javier Bértoli over 3 years ago
- Category set to Deployment
- Status changed from New to In Progress
- Target version set to 2021-07-07 sprint
Updated by Javier Bértoli over 3 years ago
- Status changed from In Progress to Feedback
Enabled the jenkins job
It uses the jenkins-image-arvados-formula-tests-{date} image created on #17844, through a cloud worker, using the label saltstack-tests.
It runs kitchen to verify the follow combinations:
$ kitchen list "debian-10-3003|centos-7-3003" Instance Driver Provisioner Verifier Transport Last Action Last Error api-debian-10-3003-0-py3 Docker SaltSolo Inspec Ssh <Not Created> <None> api-centos-7-3003-0-py3 Docker SaltSolo Inspec Ssh <Not Created> <None> workbench-debian-10-3003-0-py3 Docker SaltSolo Inspec Ssh <Not Created> <None> workbench-centos-7-3003-0-py3 Docker SaltSolo Inspec Ssh <Not Created> <None> shell-debian-10-3003-0-py3 Docker SaltSolo Inspec Ssh <Not Created> <None> shell-centos-7-3003-0-py3 Docker SaltSolo Inspec Ssh <Not Created> <None> keepstore-debian-10-3003-0-py3 Docker SaltSolo Inspec Ssh <Not Created> <None> keepstore-centos-7-3003-0-py3 Docker SaltSolo Inspec Ssh <Not Created> <None>
(run kitchen list to see all the available)
The naming of the suites follow the kitchen-ci format: suite_name-platform, which are built following the Saltstack-formulas images (for more details see the suites and platforms sections in the kitchen.yml file)
Note that some arvados components are applied and tested on the same image, to speed up testing.
Also, tests are run concurrently for the same reason.
At the moment, centos tests are failing because of #17750, #17787, which is expected.
Didn't enable the ubuntu testing because debian should cover it, but can be enabled if desired. It's just a matter of adding an entry in the regex
Updated by Peter Amstutz over 3 years ago
- Target version changed from 2021-07-07 sprint to 2021-07-21 sprint
Updated by Nico César over 3 years ago
At the end there is some extra info:
>>>>>> Converge failed on instance <workbench-centos-7-3003-0-py3>. Please see .kitchen/logs/workbench-centos-7-3003-0-py3.log for more details >>>>>> Converge failed on instance <api-centos-7-3003-0-py3>. Please see .kitchen/logs/api-centos-7-3003-0-py3.log for more details
are the .kitchen/logs/* files lost in the worker instance that runs it? if that's the case, can we add it to "workspace files" to keep them?
Updated by Nico César over 3 years ago
I also noticed that "Color ANSI Console Output" is enabled for that job, butsomehow the colors are not there and the escape characters are... is there an easy way to disable colors all together?
Updated by Javier Bértoli over 3 years ago
Nico César wrote:
I also noticed that "Color ANSI Console Output" is enabled for that job, butsomehow the colors are not there and the escape characters are... is there an easy way to disable colors all together?
There are ways and I tried a few (with --no-color, txt, yaml)
It seems to be an issue with the ANSIColor plugin, because other parts of the output are correctly colored and when the run ends OK, the output is not garbled even where it fails in the other cases.
I'll enable "yaml", as it seems to be the only one that is correctly managed by the ANSIcolor plugin.
Updated by Javier Bértoli over 3 years ago
Nico César wrote:
I also noticed that "Color ANSI Console Output" is enabled for that job, butsomehow the colors are not there and the escape characters are... is there an easy way to disable colors all together?
Kept the formula code as-is, so running the formula on a local workspace works as usual, but modified the jenkins' job configuration to add a kitchen.local.yml" file with a @yaml outputter, so it's readable in jenkins.
A weird twist is that, adding an invalid output module, produces a yaml output as the one above, but that is parseable by jenkins and, therefore, colored
Updated by Javier Bértoli over 3 years ago
Nico César wrote:
At the end there is some extra info:
[...]
are the .kitchen/logs/* files lost in the worker instance that runs it? if that's the case, can we add it to "workspace files" to keep them?
The .kitchen/logs/* files are a per-suite file with the output already seen in jenkins' output. I've used them in very rare situations, but we can add them, I guess as artifacts?
Updated by Peter Amstutz over 3 years ago
- Target version changed from 2021-07-21 sprint to 2021-08-04 sprint
Updated by Javier Bértoli over 3 years ago
Modified the job to save the logs as artifacts as suggested
Updated by Peter Amstutz over 3 years ago
- Target version changed from 2021-08-04 sprint to 2021-08-18 sprint
Updated by Javier Bértoli over 3 years ago
Added a few more changes to fix Nginx/Passenger/Arvados-{api,workbench} issues related to bundler version.
arvados-formula:commit:f3e5a4b branch fix-rvm-deps-in-centos.
Updated by