Arvados

Javier Bértoli wrote:

Fixed on 972f9d0c4, branch 18631-shell-login-sync

All the arv commands fail with bring-your-own certs. The ARVADOS_API_HOST_INSECURE=true environment variable should be set.

For example:

----------
          ID: arvados-controller-resources-virtual-machines-shell-record-cmd-run
    Function: cmd.run
        Name: arv --format=uuid \
  virtual_machine \
  create \
  --virtual-machine '{"hostname":"wardtest.arvados.org" }'

      Result: False
     Comment: Command "arv --format=uuid \
                virtual_machine \
                create \
                --virtual-machine '{"hostname":"wardtest.arvados.org" }'
              " run
     Started: 12:26:40.550445
    Duration: 833.196 ms
     Changes:   
              ----------
              pid:
                  25814
              retcode:
                  1
              stderr:
              stdout:
                  Failed to connect to Arvados API server: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

Can you fix that and I'll test again.

Should be addressed in 0e089fd82

Javier Bértoli wrote:

Should be addressed in 0e089fd82

Another bug, the ARVADOS_VIRTUAL_MACHINE_UUID field is populated with the wrong value in the cron job:

root@ip-10-253-0-192:~# crontab -l
# Lines below here are managed by Salt, do not edit
ARVADOS_API_HOST=wardtest.arvados.org:8800
ARVADOS_API_TOKEN="5sh8i643ygr18xzr9pdi5fwwgeksfq91renz9c832afllehgbg" 
ARVADOS_API_HOST_INSECURE=True
ARVADOS_VIRTUAL_MACHINE_UUID="5sh8i643ygr18xzr9pdi5fwwgeksfq91renz9c832afllehgbg" 
# SALT_CRON_IDENTIFIER:arvados-login-sync
*/2 * * * * arvados-login-sync

As you can see, that is not a UUID, it's putting the token there instead of the uuid of the machine.

After fixing the UUID field, I get

root@ip-10-253-0-192:~# arvados-login-sync
Add user admin to admin group
Remove user admin from adm group
Removing user admin from group adm
Remove user admin from dialout group
Removing user admin from group dialout
Remove user admin from cdrom group
Removing user admin from group cdrom
Remove user admin from floppy group
Removing user admin from group floppy
Remove user admin from audio group
Removing user admin from group audio
Remove user admin from dip group
Removing user admin from group dip
Remove user admin from video group
Removing user admin from group video
Remove user admin from plugdev group
Removing user admin from group plugdev
Remove user admin from netdev group
Removing user admin from group netdev
Account admin token file not found, creating new token.
Error setting token for admin: ["Forbidden (req-13zsh0hezc2051biv8xe)"]

I don't know which token is used here, but it's wrong because it's not allowed to create tokens for other users. It should be the SystemRootToken.

Javier Bértoli wrote:

Should be addressed in 0e089fd82

I tested, it does not fix it in the "bring-your-own" cerificates case.

Modified the arvados-formula's state where it creates the virtual-machines entries, to honor ARVADOS_API_HOST_INSECURE (commit:0b5a8c3:arvados-formula, branch 18631-shell-login-sync):

    arvados-sh-sn: # Create the virtual machine record
    arvados-sh-sn: arvados-controller-resources-virtual-machines-shell-record-cmd-run:
    arvados-sh-sn:   cmd.run:
    arvados-sh-sn:     - env:
    arvados-sh-sn:       - ARVADOS_API_TOKEN: "systemroottokenmushaveatleast32characters" 
    arvados-sh-sn:       - ARVADOS_API_HOST: cnts7.local:8800
    arvados-sh-sn:       - ARVADOS_API_HOST_INSECURE: True
    arvados-sh-sn:     - name: |
    arvados-sh-sn:         arv --format=uuid \
    arvados-sh-sn:           virtual_machine \
    arvados-sh-sn:           create \
    arvados-sh-sn:           --virtual-machine '{"hostname":"cnts7.local" }'
    arvados-sh-sn:     - unless: |
    arvados-sh-sn:           ARVADOS_API_TOKEN="systemroottokenmushaveatleast32characters" ARVADOS_API_HOST=cnts7.local:8800 ARVADOS_API_HOST_INSECURE=True arv --short virtual_machine list --filters '[["hostname", "=", "cnts7.local"]]' | \
    arvados-sh-sn:           /bin/grep -qE "[a-z0-9]{5}-2x53u-[a-z0-9]{15}" 
    arvados-sh-sn:     - require:
    arvados-sh-sn:       - pkg: arvados-controller-package-install-pkg-installed
    arvados-sh-sn:       - cmd: arvados-controller-service-running-service-ready-cmd-run
    arvados-sh-sn:   

Javier Bértoli wrote:

Modified the arvados-formula's state where it creates the virtual-machines entries, to honor ARVADOS_API_HOST_INSECURE (commit:0b5a8c3:arvados-formula, branch 18631-shell-login-sync):

[...]

Okay, trying with BRANCH=18631-shell-login-sync in local.params, that gets me past the bug, so that one should be merged, thanks.

Updated the provision configs to use SystemRootToken in 07440bacc

Javier Bértoli wrote:

Updated the provision configs to use SystemRootToken in 07440bacc

Confirmed that works, thanks!

testing with multihost now, installing just the shell role somehow pulls in (half of?) the api server ? And that fails of course:

$ ssh installer-test-node-2 ./provision.sh --config local.params --roles shell
Detected distro: debian

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
Hit:2 http://cdn-aws.deb.debian.org/debian bullseye InRelease
Hit:3 http://cdn-aws.deb.debian.org/debian bullseye-updates InRelease
Hit:4 http://cdn-aws.deb.debian.org/debian bullseye-backports InRelease
Hit:5 https://repo.saltproject.io/py3/debian/11/amd64/latest bullseye InRelease
Hit:6 http://apt.arvados.org/bullseye bullseye InRelease
Reading package lists...
Building dependency tree...
Reading state information...
18 packages can be upgraded. Run 'apt list --upgradable' to see them.

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.74.0-1.3+deb11u1).
git is already the newest version (1:2.30.2-1).
jq is already the newest version (1.6-2.1).
0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
/usr/bin/salt-call
Salt already installed
Cloning formulas
...locale
...nginx
...postgres
...letsencrypt
...arvados
Writing pillars and states
[INFO    ] Loading fresh modules for state activity
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/clouds'
[INFO    ] Syncing clouds for environment 'base'
[INFO    ] Loading cache from salt://_clouds, for base
[INFO    ] Caching directory '_clouds/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/beacons'
[INFO    ] Syncing beacons for environment 'base'
[INFO    ] Loading cache from salt://_beacons, for base
[INFO    ] Caching directory '_beacons/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base
[INFO    ] Caching directory '_modules/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base
[INFO    ] Caching directory '_states/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/sdb'
[INFO    ] Syncing sdb for environment 'base'
[INFO    ] Loading cache from salt://_sdb, for base
[INFO    ] Caching directory '_sdb/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base
[INFO    ] Caching directory '_grains/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[INFO    ] Syncing renderers for environment 'base'
[INFO    ] Loading cache from salt://_renderers, for base
[INFO    ] Caching directory '_renderers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[INFO    ] Syncing returners for environment 'base'
[INFO    ] Loading cache from salt://_returners, for base
[INFO    ] Caching directory '_returners/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/output'
[INFO    ] Syncing output for environment 'base'
[INFO    ] Loading cache from salt://_output, for base
[INFO    ] Caching directory '_output/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[INFO    ] Syncing utils for environment 'base'
[INFO    ] Loading cache from salt://_utils, for base
[INFO    ] Caching directory '_utils/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/log_handlers'
[INFO    ] Syncing log_handlers for environment 'base'
[INFO    ] Loading cache from salt://_log_handlers, for base
[INFO    ] Caching directory '_log_handlers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/executors'
[INFO    ] Syncing executors for environment 'base'
[INFO    ] Loading cache from salt://_executors, for base
[INFO    ] Caching directory '_executors/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/proxy'
[INFO    ] Syncing proxy for environment 'base'
[INFO    ] Loading cache from salt://_proxy, for base
[INFO    ] Caching directory '_proxy/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/engines'
[INFO    ] Syncing engines for environment 'base'
[INFO    ] Loading cache from salt://_engines, for base
[INFO    ] Caching directory '_engines/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/thorium'
[INFO    ] Syncing thorium for environment 'base'
[INFO    ] Loading cache from salt://_thorium, for base
[INFO    ] Caching directory '_thorium/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/serializers'
[INFO    ] Syncing serializers for environment 'base'
[INFO    ] Loading cache from salt://_serializers, for base
[INFO    ] Caching directory '_serializers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/matchers'
[INFO    ] Syncing matchers for environment 'base'
[INFO    ] Loading cache from salt://_matchers, for base
[INFO    ] Caching directory '_matchers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/pillar'
[INFO    ] Syncing pillar for environment 'base'
[INFO    ] Loading cache from salt://_pillar, for base
[INFO    ] Caching directory '_pillar/' for environment 'base'
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [locale_pkgs] at time 15:47:43.167777
[INFO    ] Executing state pkg.installed for [locale_pkgs]
[INFO    ] Executing command dpkg-query in directory '/root'
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [locale_pkgs] at time 15:47:43.193615 (duration_in_ms=25.838)
[INFO    ] Running state [en_US.UTF-8 UTF-8] at time 15:47:43.194902
[INFO    ] Executing state locale.present for [en_US.UTF-8 UTF-8]
[INFO    ] Executing command 'locale' in directory '/root'
[INFO    ] Locale en_US.UTF-8 UTF-8 is already present
[INFO    ] Completed state [en_US.UTF-8 UTF-8] at time 15:47:43.200810 (duration_in_ms=5.907)
[INFO    ] Running state [dbus] at time 15:47:43.201096
[INFO    ] Executing state pkg.installed for [dbus]
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [dbus] at time 15:47:43.208732 (duration_in_ms=7.637)
[INFO    ] Running state [dbus] at time 15:47:43.211205
[INFO    ] Executing state service.running for [dbus]
[INFO    ] Executing command systemctl in directory '/root'
[INFO    ] Executing command systemctl in directory '/root'
[INFO    ] Executing command systemctl in directory '/root'
[INFO    ] The service dbus is already running
[INFO    ] Completed state [dbus] at time 15:47:43.239242 (duration_in_ms=28.038)
[INFO    ] Running state [en_US.UTF-8] at time 15:47:43.240640
[INFO    ] Executing state locale.system for [en_US.UTF-8]
[INFO    ] System locale en_US.UTF-8 already set
[INFO    ] Completed state [en_US.UTF-8] at time 15:47:43.290550 (duration_in_ms=49.91)
[INFO    ] Running state [db.xxxx1.arvadosapi.com] at time 15:47:43.292613
[INFO    ] Executing state host.present for [db.xxxx1.arvadosapi.com]
[INFO    ] Host db.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [db.xxxx1.arvadosapi.com] at time 15:47:43.293879 (duration_in_ms=1.266)
[INFO    ] Running state [database.xxxx1.arvadosapi.com] at time 15:47:43.294055
[INFO    ] Executing state host.present for [database.xxxx1.arvadosapi.com]
[INFO    ] Host database.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [database.xxxx1.arvadosapi.com] at time 15:47:43.294735 (duration_in_ms=0.679)
[INFO    ] Running state [xxxx1.arvadosapi.com] at time 15:47:43.294898
[INFO    ] Executing state host.present for [xxxx1.arvadosapi.com]
[INFO    ] Host xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [xxxx1.arvadosapi.com] at time 15:47:43.295505 (duration_in_ms=0.608)
[INFO    ] Running state [ws.xxxx1.arvadosapi.com] at time 15:47:43.295663
[INFO    ] Executing state host.present for [ws.xxxx1.arvadosapi.com]
[INFO    ] Host ws.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [ws.xxxx1.arvadosapi.com] at time 15:47:43.296274 (duration_in_ms=0.61)
[INFO    ] Running state [workbench.xxxx1.arvadosapi.com] at time 15:47:43.296436
[INFO    ] Executing state host.present for [workbench.xxxx1.arvadosapi.com]
[INFO    ] Host workbench.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [workbench.xxxx1.arvadosapi.com] at time 15:47:43.297063 (duration_in_ms=0.627)
[INFO    ] Running state [workbench2.xxxx1.arvadosapi.com] at time 15:47:43.297218
[INFO    ] Executing state host.present for [workbench2.xxxx1.arvadosapi.com]
[INFO    ] Host workbench2.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [workbench2.xxxx1.arvadosapi.com] at time 15:47:43.297816 (duration_in_ms=0.597)
[INFO    ] Running state [keep.xxxx1.arvadosapi.com] at time 15:47:43.297945
[INFO    ] Executing state host.present for [keep.xxxx1.arvadosapi.com]
[INFO    ] Host keep.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [keep.xxxx1.arvadosapi.com] at time 15:47:43.298552 (duration_in_ms=0.607)
[INFO    ] Running state [download.xxxx1.arvadosapi.com] at time 15:47:43.298678
[INFO    ] Executing state host.present for [download.xxxx1.arvadosapi.com]
[INFO    ] Host download.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [download.xxxx1.arvadosapi.com] at time 15:47:43.299279 (duration_in_ms=0.6)
[INFO    ] Running state [collections.xxxx1.arvadosapi.com] at time 15:47:43.299408
[INFO    ] Executing state host.present for [collections.xxxx1.arvadosapi.com]
[INFO    ] Host collections.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [collections.xxxx1.arvadosapi.com] at time 15:47:43.299983 (duration_in_ms=0.574)
[INFO    ] Running state [webshell.xxxx1.arvadosapi.com] at time 15:47:43.300111
[INFO    ] Executing state host.present for [webshell.xxxx1.arvadosapi.com]
[INFO    ] Host webshell.xxxx1.arvadosapi.com (10.253.0.96) already present
[INFO    ] Completed state [webshell.xxxx1.arvadosapi.com] at time 15:47:43.300695 (duration_in_ms=0.583)
[INFO    ] Running state [shell.xxxx1.arvadosapi.com] at time 15:47:43.300824
[INFO    ] Executing state host.present for [shell.xxxx1.arvadosapi.com]
[INFO    ] Host shell.xxxx1.arvadosapi.com (10.253.0.34) already present
[INFO    ] Completed state [shell.xxxx1.arvadosapi.com] at time 15:47:43.301402 (duration_in_ms=0.578)
[INFO    ] Running state [keep0.xxxx1.arvadosapi.com] at time 15:47:43.301529
[INFO    ] Executing state host.present for [keep0.xxxx1.arvadosapi.com]
[INFO    ] Host keep0.xxxx1.arvadosapi.com (10.253.0.34) already present
[INFO    ] Completed state [keep0.xxxx1.arvadosapi.com] at time 15:47:43.303500 (duration_in_ms=1.97)
[INFO    ] Running state [keep1.xxxx1.arvadosapi.com] at time 15:47:43.303631
[INFO    ] Executing state host.present for [keep1.xxxx1.arvadosapi.com]
[INFO    ] Host keep1.xxxx1.arvadosapi.com (10.0.0.4) already present
[INFO    ] Completed state [keep1.xxxx1.arvadosapi.com] at time 15:47:43.304198 (duration_in_ms=0.567)
[INFO    ] Running state [/usr/share/keyrings/arvados-archive-keyring.gpg] at time 15:47:43.306064
[INFO    ] Executing state file.managed for [/usr/share/keyrings/arvados-archive-keyring.gpg]
[INFO    ] File /usr/share/keyrings/arvados-archive-keyring.gpg is in the correct state
[INFO    ] Completed state [/usr/share/keyrings/arvados-archive-keyring.gpg] at time 15:47:43.401820 (duration_in_ms=95.755)
[INFO    ] Running state [/etc/apt/sources.list.d/arvados.list] at time 15:47:43.402484
[INFO    ] Executing state file.managed for [/etc/apt/sources.list.d/arvados.list]
[INFO    ] File /etc/apt/sources.list.d/arvados.list is in the correct state
[INFO    ] Completed state [/etc/apt/sources.list.d/arvados.list] at time 15:47:43.404652 (duration_in_ms=2.168)
[INFO    ] Running state [sudo] at time 15:47:43.404822
[INFO    ] Executing state pkg.installed for [sudo]
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [sudo] at time 15:47:43.412254 (duration_in_ms=7.432)
[INFO    ] Running state [/etc/sudoers.d/arvados_passwordless] at time 15:47:43.412827
[INFO    ] Executing state file.managed for [/etc/sudoers.d/arvados_passwordless]
[INFO    ] File /etc/sudoers.d/arvados_passwordless exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/sudoers.d/arvados_passwordless] at time 15:47:43.414034 (duration_in_ms=1.207)
[INFO    ] Running state [arvados-server] at time 15:47:43.414220
[INFO    ] Executing state pkg.installed for [arvados-server]
[INFO    ] Executing command apt-get in directory '/root'
[INFO    ] Executing command apt-cache in directory '/root'
[INFO    ] All specified packages are already installed and are at the desired version
[INFO    ] Completed state [arvados-server] at time 15:47:44.482385 (duration_in_ms=1068.165)
[INFO    ] Running state [/etc/arvados/config.yml] at time 15:47:44.483282
[INFO    ] Executing state file.managed for [/etc/arvados/config.yml]
[INFO    ] File /etc/arvados/config.yml is in the correct state
[INFO    ] Completed state [/etc/arvados/config.yml] at time 15:47:44.512678 (duration_in_ms=29.396)
[INFO    ] Running state [ruby] at time 15:47:44.512863
[INFO    ] Executing state pkg.installed for [ruby]
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [ruby] at time 15:47:44.520081 (duration_in_ms=7.218)
[INFO    ] Running state [arvados-api-package-install-gems-deps-pkg-installed] at time 15:47:44.520243
[INFO    ] Executing state pkg.installed for [arvados-api-package-install-gems-deps-pkg-installed]
[INFO    ] Executing command git in directory '/root'
[INFO    ] Executing command 'test' in directory '/root'
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [arvados-api-package-install-gems-deps-pkg-installed] at time 15:47:44.818260 (duration_in_ms=298.014)
[INFO    ] Running state [arvados-cli] at time 15:47:44.819701
[INFO    ] Executing state gem.installed for [arvados-cli]
[INFO    ] Executing command gem in directory '/root'
[INFO    ] Gem is already installed.
[INFO    ] Completed state [arvados-cli] at time 15:47:44.986238 (duration_in_ms=166.536)
[INFO    ] Running state [arvados-api-server] at time 15:47:44.987368
[INFO    ] Executing state pkg.installed for [arvados-api-server]
[INFO    ] Executing command apt-get in directory '/root'
[INFO    ] Executing command apt-cache in directory '/root'
[INFO    ] All specified packages are already installed and are at the desired version
[INFO    ] Completed state [arvados-api-server] at time 15:47:46.012112 (duration_in_ms=1024.744)
[INFO    ] Running state [nginx] at time 15:47:46.013288
[INFO    ] Executing state service.running for [nginx]
[INFO    ] Executing command systemctl in directory '/root'
[ERROR   ] The named service nginx is not available
[INFO    ] Completed state [nginx] at time 15:47:46.024420 (duration_in_ms=11.131)
local:
----------
          ID: locale_pkgs
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 15:47:43.167777
    Duration: 25.838 ms
     Changes:   
----------
          ID: locale_present_en_US_UTF-8_UTF-8
    Function: locale.present
        Name: en_US.UTF-8 UTF-8
      Result: True
     Comment: Locale en_US.UTF-8 UTF-8 is already present
     Started: 15:47:43.194903
    Duration: 5.907 ms
     Changes:   
----------
          ID: dbus_for_locale
    Function: pkg.installed
        Name: dbus
      Result: True
     Comment: All specified packages are already installed
     Started: 15:47:43.201095
    Duration: 7.637 ms
     Changes:   
----------
          ID: dbus_for_locale
    Function: service.running
        Name: dbus
      Result: True
     Comment: The service dbus is already running
     Started: 15:47:43.211204
    Duration: 28.038 ms
     Changes:   
----------
          ID: locale_default
    Function: locale.system
        Name: en_US.UTF-8
      Result: True
     Comment: System locale en_US.UTF-8 already set
     Started: 15:47:43.240640
    Duration: 49.91 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_database_host_present
    Function: host.present
        Name: db.xxxx1.arvadosapi.com
      Result: True
     Comment: Host db.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.292613
    Duration: 1.266 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_database_host_present
    Function: host.present
        Name: database.xxxx1.arvadosapi.com
      Result: True
     Comment: Host database.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.294056
    Duration: 0.679 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_api_host_present
    Function: host.present
        Name: xxxx1.arvadosapi.com
      Result: True
     Comment: Host xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.294897
    Duration: 0.608 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_websocket_host_present
    Function: host.present
        Name: ws.xxxx1.arvadosapi.com
      Result: True
     Comment: Host ws.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.295664
    Duration: 0.61 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_workbench_host_present
    Function: host.present
        Name: workbench.xxxx1.arvadosapi.com
      Result: True
     Comment: Host workbench.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.296436
    Duration: 0.627 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_workbench2_host_present
    Function: host.present
        Name: workbench2.xxxx1.arvadosapi.com
      Result: True
     Comment: Host workbench2.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.297219
    Duration: 0.597 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_keepproxy_host_present
    Function: host.present
        Name: keep.xxxx1.arvadosapi.com
      Result: True
     Comment: Host keep.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.297945
    Duration: 0.607 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_keepweb_host_present
    Function: host.present
        Name: download.xxxx1.arvadosapi.com
      Result: True
     Comment: Host download.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.298679
    Duration: 0.6 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_keepweb_host_present
    Function: host.present
        Name: collections.xxxx1.arvadosapi.com
      Result: True
     Comment: Host collections.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.299409
    Duration: 0.574 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_webshell_host_present
    Function: host.present
        Name: webshell.xxxx1.arvadosapi.com
      Result: True
     Comment: Host webshell.xxxx1.arvadosapi.com (10.253.0.96) already present
     Started: 15:47:43.300112
    Duration: 0.583 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_shell_host_present
    Function: host.present
        Name: shell.xxxx1.arvadosapi.com
      Result: True
     Comment: Host shell.xxxx1.arvadosapi.com (10.253.0.34) already present
     Started: 15:47:43.300824
    Duration: 0.578 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_keep0_host_present
    Function: host.present
        Name: keep0.xxxx1.arvadosapi.com
      Result: True
     Comment: Host keep0.xxxx1.arvadosapi.com (10.253.0.34) already present
     Started: 15:47:43.301530
    Duration: 1.97 ms
     Changes:   
----------
          ID: extra_extra_hosts_entries_etc_hosts_keep1_host_present
    Function: host.present
        Name: keep1.xxxx1.arvadosapi.com
      Result: True
     Comment: Host keep1.xxxx1.arvadosapi.com (10.0.0.4) already present
     Started: 15:47:43.303631
    Duration: 0.567 ms
     Changes:   
----------
          ID: arvados-repo-install-pkgrepo-keyring-managed
    Function: file.managed
        Name: /usr/share/keyrings/arvados-archive-keyring.gpg
      Result: True
     Comment: File /usr/share/keyrings/arvados-archive-keyring.gpg is in the correct state
     Started: 15:47:43.306065
    Duration: 95.755 ms
     Changes:   
----------
          ID: arvados-repo-install-file-managed
    Function: file.managed
        Name: /etc/apt/sources.list.d/arvados.list
      Result: True
     Comment: File /etc/apt/sources.list.d/arvados.list is in the correct state
     Started: 15:47:43.402484
    Duration: 2.168 ms
     Changes:   
----------
          ID: extra_shell_sudo_passwordless_sudo_pkg_installed
    Function: pkg.installed
        Name: sudo
      Result: True
     Comment: All specified packages are already installed
     Started: 15:47:43.404822
    Duration: 7.432 ms
     Changes:   
----------
          ID: extra_shell_sudo_passwordless_config_file_managed
    Function: file.managed
        Name: /etc/sudoers.d/arvados_passwordless
      Result: True
     Comment: File /etc/sudoers.d/arvados_passwordless exists with proper permissions. No changes made.
     Started: 15:47:43.412827
    Duration: 1.207 ms
     Changes:   
----------
          ID: arvados-config-package-install-pkg-installed
    Function: pkg.installed
        Name: arvados-server
      Result: True
     Comment: All specified packages are already installed and are at the desired version
     Started: 15:47:43.414220
    Duration: 1068.165 ms
     Changes:   
----------
          ID: arvados-config-file-file-managed
    Function: file.managed
        Name: /etc/arvados/config.yml
      Result: True
     Comment: File /etc/arvados/config.yml is in the correct state
     Started: 15:47:44.483282
    Duration: 29.396 ms
     Changes:   
----------
          ID: arvados-ruby-package-install-ruby-pkg-installed
    Function: pkg.installed
        Name: ruby
      Result: True
     Comment: All specified packages are already installed
     Started: 15:47:44.512863
    Duration: 7.218 ms
     Changes:   
----------
          ID: arvados-api-package-install-gems-deps-pkg-installed
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 15:47:44.520246
    Duration: 298.014 ms
     Changes:   
----------
          ID: arvados-api-package-install-gem-arvados-cli-installed
    Function: gem.installed
        Name: arvados-cli
      Result: True
     Comment: Gem is already installed.
     Started: 15:47:44.819702
    Duration: 166.536 ms
     Changes:   
----------
          ID: arvados-api-package-install-pkg-installed
    Function: pkg.installed
        Name: arvados-api-server
      Result: True
     Comment: All specified packages are already installed and are at the desired version
     Started: 15:47:44.987368
    Duration: 1024.744 ms
     Changes:   
----------
          ID: arvados-api-service-running-service-running
    Function: service.running
        Name: nginx
      Result: False
     Comment: The named service nginx is not available
     Started: 15:47:46.013289
    Duration: 11.131 ms
     Changes:   

Summary for local
-------------
Succeeded: 28
Failed:     1
-------------
Total states run:     29
Total run time:    2.839 s

at ad9dfa8583197097caf3d2fb41771e9f09e55252 no longer errors out, but the cron job is not installed:

root@ip-10-253-0-27:~# crontab -l
no crontab for root

As discussed, moved the resources/virtual_machines.sls state file as a sub-component of the shell states, in commit:eef0881:arvados-formula, branch 18631-shell-login-sync

Updated the provision config examples to set up the login-sync cron correctly, e4aefa613, branch 18631-shell-login-sync

Javier Bértoli wrote:

As discussed, moved the resources/virtual_machines.sls state file as a sub-component of the shell states, in commit:eef0881:arvados-formula, branch 18631-shell-login-sync

Updated the provision config examples to set up the login-sync cron correctly, e4aefa613, branch 18631-shell-login-sync

Testing with the latest from 18631-shell-login-sync and

BRANCH=18631-shell-login-sync

in local.params.

The cron job is installed now but it does not work because it is missing a path:

...
*/2 * * * * arvados-login-sync

And in root's mailbox:

X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <E1nc5U1-0007U9-42@ip-10-253-0-178.arvadosapi.com>
Date: Wed, 06 Apr 2022 13:14:01 +0000

/bin/sh: 1: arvados-login-sync: not found

This needs to be updated to /usr/local/bin/arvados-login-sync (or, the PATH var could be set in the cron job).

Otherwise, LGTM thanks.