Arvados

As of commit:

# file keepstore*arm64*
keepstore_2.4.0~dev20211222142811-1_arm64.deb: Debian binary package (format 2.0), with control.tar.gz, data compression gz

I booted a t4g-micro instance to verify that the package installs and that the binary is executable:

# dpkg -i keepstore_2.4.0~dev20211222142811-1_arm64.deb 
Selecting previously unselected package keepstore.
(Reading database ... 29276 files and directories currently installed.)
Preparing to unpack keepstore_2.4.0~dev20211222142811-1_arm64.deb ...
Unpacking keepstore (2.4.0~dev20211222142811-1) ...
Setting up keepstore (2.4.0~dev20211222142811-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/keepstore.service → /lib/systemd/system/keepstore.service.
Assertion failed on job for keepstore.service.

#  journalctl -u keepstore
-- Journal begins at Wed 2021-12-22 17:56:27 UTC, ends at Wed 2021-12-22 18:01:16 UTC. --
Dec 22 17:58:17 ip-10-254-254-166 systemd[1]: keepstore.service: Starting requested but asserts failed.
Dec 22 17:58:17 ip-10-254-254-166 systemd[1]: Assertion failed for Arvados Keep Storage Daemon.

# keepstore
{"error":"open /etc/arvados/config.yml: no such file or directory","level":"error","msg":"exiting","time":"2021-12-22T17:58:37.675162849Z"}

# file /usr/bin/keepstore
/usr/bin/keepstore: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=7c35b7e1fdc7e637650f114f4e2571507f920fe6, for GNU/Linux 3.7.0, not stripped

Switching from therubyracer to mini_racer in f4593eec39e8d2d4804c0a0197b510cfd760087d on branch 17417-switch-to-mini_racer. This also removes the leftover `less` and `less-rails` gems from WB1's Gemfile; they were no longer used.

Tests passed at developer-run-tests: #2850

Just as a test, removed mini_racer from both RailsAPI and WB1 at 7085a21aa - branch 17417-goodbye-to-mini_racer
Running tests at: developer-run-tests: #2852

Lucas Di Pentima wrote:

Just as a test, removed mini_racer from both RailsAPI and WB1 at 7085a21aa - branch 17417-goodbye-to-mini_racer
Running tests at: developer-run-tests: #2852

Those tests passed, great! It seems there may be a few more gems in the API server Gemfile that are superfluous, I've pushed 9946e935db217d6d470bd2aea79a49b155d982ac on branch 17417-goodbye-to-mini_racer. I've tested locally that the arvados-api-server package still builds and installs.

Running tests at developer-run-tests: #2854 . Re-running the wb1 integration tests at developer-run-tests-apps-workbench-integration: #3034 , but that's an unrelated failure.

9946e935db217d6d470bd2aea79a49b155d982ac LGTM.

Lucas Di Pentima wrote:

9946e935db217d6d470bd2aea79a49b155d982ac LGTM.

Thanks, merged!

a4a1420766c6e2e84e61f1d5e8cbb319521af31e on branch 17417-add-arm64

• adds cross-compilation to arm64 for our golang packages, when running on amd64, on debian11 only
• adds native build on arm64 for all our packages except for workbench, on debian11 only

Ward Vandewege wrote:

Lucas Di Pentima wrote:

9946e935db217d6d470bd2aea79a49b155d982ac LGTM.

Thanks, merged!

And now that it's deployed to 9tee4, ce8i5 and tordo, it seems we really do need a JavaScript runtime for WB1 (not api server!). Error from tordo:

Could not find a JavaScript runtime. See https://github.com/rails/execjs for a list of available runtimes. (ExecJS::RuntimeUnavailable)
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/runtimes.rb:58:in `autodetect'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs.rb:5:in `<module:ExecJS>'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs.rb:4:in `<top (required)>'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/coffee-script-2.4.1/lib/coffee_script.rb:1:in `<top (required)>'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/coffee-script-2.4.1/lib/coffee-script.rb:1:in `<top (required)>'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/coffee-rails-4.2.2/lib/coffee-rails.rb:1:in `<top (required)>'
  /var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:66:in `require'
  /var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:66:in `block (2 levels) in require'
  /var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:61:in `each'
  /var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:61:in `block in require'
  /var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:50:in `each'
  /var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:50:in `require'
  /var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler.rb:174:in `require'
  /var/www/arvados-workbench/current/config/application.rb:22:in `<top (required)>'
  /var/www/arvados-workbench/current/config/environment.rb:6:in `require_relative'
  /var/www/arvados-workbench/current/config/environment.rb:6:in `<top (required)>'
  config.ru:7:in `require'
  config.ru:7:in `block in <main>'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:125:in `instance_eval'
  /var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:125:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:101:in `eval'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:101:in `preload_app'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:189:in `block in <module:App>'
  /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:397:in `run_block_and_record_step_progress'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:188:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'

Weirdly, on 9tee4 there is no error. Why? The same version of arvados-workbench appears to be installed:

workbench.tordo:~# dpkg -l |grep arvados-workbench
ii  arvados-workbench                       2.4.0~dev20211223212204-1     amd64        Arvados Workbench - Arvados is a free and open source platform for big data science.

9tee4:~# dpkg -l |grep arvados-work
ii  arvados-workbench                     2.4.0~dev20211223212204-1      amd64        Arvados Workbench - Arvados is a free and open source platform for big data science.

Both 9tee4 and tordo are on Debian 10.

Hmm, I built a wb1 package with mini_racer added to the Gemfile, but I get the same error, even trying to install it on workbench.tordo:

dpkg -i /root/arvados-workbench_2.4.0~dev20211223212204-1_amd64.deb 
(Reading database ... 68123 files and directories currently installed.)
Preparing to unpack .../arvados-workbench_2.4.0~dev20211223212204-1_amd64.deb ...
Unpacking arvados-workbench (2.4.0~dev20211223212204-1) over (2.4.0~dev20211223212204-1) ...
Setting up arvados-workbench (2.4.0~dev20211223212204-1) ...

Assumption: nginx is configured to serve Rails from
            /var/www/arvados-workbench/current
Assumption: nginx and passenger run as www-data

Creating symlinks to configuration in /etc/arvados/workbench ...... done.
Running bundle config set --local path /var/www/arvados-workbench/shared/vendor_bundle... done.
Running bundle install... done.
Ensuring directory and file permissions ...... done.
Checking configuration for completeness...rake aborted!
ExecJS::RuntimeUnavailable: Could not find a JavaScript runtime. See https://github.com/rails/execjs for a list of available runtimes.
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/runtimes.rb:58:in `autodetect'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs.rb:5:in `<module:ExecJS>'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs.rb:4:in `<top (required)>'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/coffee-script-2.4.1/lib/coffee_script.rb:1:in `<top (required)>'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/coffee-script-2.4.1/lib/coffee-script.rb:1:in `<top (required)>'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
/var/www/arvados-workbench/shared/vendor_bundle/ruby/2.5.0/gems/coffee-rails-4.2.2/lib/coffee-rails.rb:1:in `<top (required)>'
/var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:66:in `require'
/var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:66:in `block (2 levels) in require'
/var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:61:in `each'
/var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:61:in `block in require'
/var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:50:in `each'
/var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler/runtime.rb:50:in `require'
/var/lib/gems/2.5.0/gems/bundler-2.2.19/lib/bundler.rb:174:in `require'
/var/www/arvados-workbench/current/config/application.rb:22:in `<top (required)>'
/var/www/arvados-workbench/current/Rakefile:9:in `require'
/var/www/arvados-workbench/current/Rakefile:9:in `<top (required)>'
(See full trace by running task with --trace)
 failed.

PLEASE NOTE:

The arvados-workbench package was not configured completely because
/etc/arvados/config.yml needs some tweaking.
Please refer to the documentation at
<http://doc.arvados.org/install/install-workbench-app.html#configure> for more details.

When config.yml has been modified,
reconfigure or reinstall this package.

Something about the system on tordo (and ce8i5) is causing this. It's not happening on 9tee4 and in our test suite, which installs the package in a clean docker environment, and that does not fail, cf. build-packages-debian10: #1021 /consoleFull:

16:58:39 Setting up arvados-workbench (2.4.0~dev20211223212204-1) ...
16:58:39 
16:58:39 WARNING: Web service (Nginx or Apache) not found.
16:58:39 
16:58:39 To override, set the WEB_SERVICE environment variable to the name of the service
16:58:39 hosting the Rails server.
16:58:39 
16:58:39 For Debian-based systems, then reconfigure this package with dpkg-reconfigure.
16:58:39 
16:58:39 For RPM-based systems, then reinstall this package.
16:58:39 
16:58:39 
16:58:39 Assumption:  is configured to serve Rails from
16:58:39             /var/www/arvados-workbench/current
16:58:39 Assumption:  and passenger run as www-data
16:58:39 
16:58:39 Creating symlinks to configuration in /etc/arvados/workbench ...... done.
16:58:40 Running bundle config set --local path /var/www/arvados-workbench/shared/vendor_bundle... done.
16:58:40 Running bundle install... done.
17:00:29 Ensuring directory and file permissions ...... done.
17:00:29 
17:00:29 PLEASE NOTE:
17:00:29 
17:00:29 The arvados-workbench package was not configured completely because
17:00:29 /etc/arvados/config.yml needs some tweaking.
17:00:29 Please refer to the documentation at
17:00:29 <http://doc.arvados.org/install/install-workbench-app.html#configure> for more details.
17:00:29 
17:00:29 When config.yml has been modified,
17:00:29 reconfigure or reinstall this package.
17:00:29 

Though... maybe that test is not as thorough as it should be? It doesn't run the bit that fails on install (`Checking configuration for completeness...rake aborted!`) because of missing config.

Okay so now we have a few mysteries:

• why does this error not happen in the package testing stage on jenkins (or locally)
• why does this error happen with a custom-built wb1 package that does have mini_racer in it.
• why does this error not happen on 9tee4? -> probably because the nodejs package is installed there (cf. https://stackoverflow.com/questions/8059332/rails-could-not-find-a-javascript-runtime-see-https-github-com-sstephenson-e)

Tackling the package testing first. Testing manually with a config.yml in place in the wb1 test container:

--- 
Clusters:
  xxxxx:
    Services:
      Workbench1:
        ExternalURL: "https://workbench.xxxxx.example.com" 
      WebDAV:
        ExternalURL: https://*.collections.xxxxx.example.com/
      WebDAVDownload:
        ExternalURL: https://download.xxxxx.example.com
    ManagementToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    SystemRootToken: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    Collections:
      BlobSigningKey: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    Workbench:
      SecretKeyBase: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    Users:
      AutoAdminFirstUser: true

I can reproduce the error. So, that's an easy addition to our build/test scripts.

• why are we seeing this in a Gemfile with mini_racer in it?

Looks like this could have something to do with the version of gcc used to build mini_racer? Cf. https://github.com/rubyjs/mini_racer/issues/176. But, our build image for debian11 uses GCC 10:

root@1448f56594e5:/# dpkg -l |grep gcc
ii  gcc                              4:10.2.1-1                     amd64        GNU C compiler
ii  gcc-10                           10.2.1-6                       amd64        GNU C compiler
ii  gcc-10-base:amd64                10.2.1-6                       amd64        GCC, the GNU Compiler Collection (base package)
ii  gcc-9-base:amd64                 9.3.0-3                        amd64        GCC, the GNU Compiler Collection (base package)
ii  libgcc-10-dev:amd64              10.2.1-6                       amd64        GCC support library (development files)
ii  libgcc-s1:amd64                  10.2.1-6                       amd64        GCC support library

And debian10 uses 8.3:

root@8714427f8f84:/# dpkg -l |grep gcc
ii  gcc                              4:8.3.0-1                   amd64        GNU C compiler
ii  gcc-8                            8.3.0-6                     amd64        GNU C compiler
ii  gcc-8-base:amd64                 8.3.0-6                     amd64        GCC, the GNU Compiler Collection (base package)
ii  libgcc-8-dev:amd64               8.3.0-6                     amd64        GCC support library (development files)
ii  libgcc1:amd64                    1:8.3.0-6                   amd64        GCC support library

Okay, https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/1874 was illuminating. We have several gems that depend on ExecJS (coffee-script, uglifier, autoprefixer-rails). We only include a Javascript runtime at asset build time, and I think that is the only time we need it (but, those gems that depend on it are included not just in that scope, bug?). Yeah, moving `coffee-script` inside the `group :assets` allows wb1 to start up normally.

Alternatively, setting the EXECJS_RUNTIME=Disabled env var stops the error, e.g. in the postinst rails package script, but also if we set it in the nginx config for the wb1 passenger, as I just did experimentally on Tordo. Ideally we could drop it in an initializer (e.g. config/boot.rb does the trick - except that that also disabled execjs during asset generation, and that's not good!).

The question remains - why is this necessary when we use mini_racer but not when we use therubyracer?

IMO Coffeescript is not providing enough value to justify extra time troubleshooting things that might be related to the coffee-rails gem, even if they turn out not to be. How about we eliminate this by copying the converted JS out of the compiled JS on a production instance, renaming the files to .js, and deleting the coffee-rails gem from Gemfile entirely?

source:apps/workbench/app/assets/javascripts/bootstrap.js.coffee compiles to this

(function() {
  jQuery(function() {
    $("a[rel=popover]").popover();
    $(".tooltip").tooltip();
    return $("a[rel=tooltip]").tooltip();
  });
}).call(this);

source:apps/workbench/app/assets/javascripts/keep_disks.js.coffee compiles to this

(function() {
  var cache_age_axis_label, cache_age_hover, cache_age_in_days, float_as_percentage;

  cache_age_in_days = function(milliseconds_age) {
    var ONE_DAY;
    ONE_DAY = 1000 * 60 * 60 * 24;
    return milliseconds_age / ONE_DAY;
  };

  cache_age_hover = function(milliseconds_age) {
    return 'Cache age ' + cache_age_in_days(milliseconds_age).toFixed(1) + ' days.';
  };

  cache_age_axis_label = function(milliseconds_age) {
    return cache_age_in_days(milliseconds_age).toFixed(0) + ' days';
  };

  float_as_percentage = function(proportion) {
    return (proportion.toFixed(4) * 100) + '%';
  };

  $.renderHistogram = function(histogram_data) {
    return Morris.Area({
      element: 'cache-age-vs-disk-histogram',
      pointSize: 0,
      lineWidth: 0,
      data: histogram_data,
      xkey: 'age',
      ykeys: ['persisted', 'cache'],
      labels: ['Persisted Storage Disk Utilization', 'Cached Storage Disk Utilization'],
      ymax: 1,
      ymin: 0,
      xLabelFormat: cache_age_axis_label,
      yLabelFormat: float_as_percentage,
      dateFormat: cache_age_hover
    });
  };

}).call(this);

Tom Clegg wrote:

IMO Coffeescript is not providing enough value to justify extra time troubleshooting things that might be related to the coffee-rails gem, even if they turn out not to be. How about we eliminate this by copying the converted JS out of the compiled JS on a production instance, renaming the files to .js, and deleting the coffee-rails gem from Gemfile entirely?

Yes, that's great. I've done so in 0c14b79d003d6e1fda00cea3dcbdfca3b6d31014 on branch 17417-fix-wb1. This branch also adds more of a config.yml file to the wb1 package testing container, which means we will trap more types of early gem/dependency errors.

Tests passed at developer-run-tests: #2857

That 17417-fix-wb1 branch is ready for review.

17417-fix-wb1 LGTM, thanks!

Tom Clegg wrote:

17417-fix-wb1 LGTM, thanks!

Merged, thanks!

Ready for review at 56c4d0c08266cacbca73e77aa82939e00a0bb69e on branch 17417-add-arm64. This branch also does some refactoring and cleanup of our build scripts. A developer package build completed without error at developer-build-packages-multijob: #13 .

In addition to the normal set of packages, it is expected to generate cross-compiled arm64 versions of our golang packages for debian11 (only, because of debian bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983477). See also the note on 6e14b7d45fb47a654966b528ede41add437215e0

Some comments:

• A suggestion for branches with lots of commits: for easier reviewing I think it's better to rebase to main instead of merging main from time to time. Personally I like to look at individual commits to be able to follow the thought process and then do a final pass to the complete diff against main, and it seems it isn't possible when unrelated merge commits are present, because those changes get included too.
• At file build/run-build-test-packages-one-target.sh
  • Line 15: the variable $ONLY_BUILD gets replaced with empty string when printing the helpmessage
  • It's a wrapper for build/run-build-packages-one-target.sh, so should it also accept --arch?
  • While doing some tests to see how cross-arch package building fails on debian10, it seems that it tries to do it instead of giving the "cannot do it right now" message:

$ WORKSPACE=~/test-arvados/ ./run-build-packages-one-target.sh --target debian10 --arch arm64 --only-build arvados-server
~/test-arvados/build/package-build-dockerfiles ~/test-arvados/build
wget -cqO common-generated/go1.17.1.linux-amd64.tar.gz https://dl.google.com/go/go1.17.1.linux-amd64.tar.gz
wget -cqO common-generated/node-v10.23.1-linux-x64.tar.xz https://nodejs.org/dist/v10.23.1/node-v10.23.1-linux-x64.tar.xz
wget -cqO common-generated/mpapis.asc https://rvm.io/mpapis.asc
wget -cqO common-generated/pkuczynski.asc https://rvm.io/pkuczynski.asc
test -d debian10/generated || mkdir debian10/generated
cp -f -rlt debian10/generated common-generated/*
debian10
Sending build context to Docker daemon  277.1MB
Step 1/25 : ARG HOSTTYPE
Step 2/25 : FROM debian:buster as build_x86_64
 ---> 8a94f77c4ac3
Step 3/25 : ONBUILD ADD generated/go1.17.1.linux-amd64.tar.gz /usr/local/
[...]
START: build packages on arvados/build:debian10
Package arvados-server_2.4.0~dev20220105213852-1_arm64.deb not found, building
Building deb (arm64) package for arvados-server from cmd/arvados-server
# runtime/cgo
cgo: C compiler "aarch64-linux-gnu-gcc" not found: exec: "aarch64-linux-gnu-gcc": executable file not found in $PATH
Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
Require just the needed backports instead, or 'backports/latest'.

Error: /root/go/bin/linux_arm64/arvados-server=/usr/bin/arvados-server: Unable to figure out package name from fpm results:

{:timestamp=>"2022-01-07T14:17:01.991953+0000", :message=>"Invalid package configuration: Cannot chdir to '/root/go/bin/linux_arm64'. Does it exist?", :level=>:error}

fpm -s dir -t deb -n arvados-server --maintainer Arvados Package Maintainers <packaging@arvados.org> --vendor The Arvados Project -v 2.4.0~dev20220105213852 --iteration 1 --url=https://arvados.org --license=GNU Affero General Public License, version 3.0 --description=Arvados server daemons -aarm64 /arvados/agpl-3.0.txt=/usr/share/doc/arvados-server/agpl-3.0.txt /root/go/bin/linux_arm64/arvados-server=/usr/bin/arvados-server

ERROR: build packages on arvados/build:debian10 failed with exit status 1

• It seems that -j is not necessary, while testing I got the message: "Found user configured '-j' flag in 'rvm_make_flags', please note that RVM can detect number of CPU threads and set the '-j' flag automatically if you do not set it."

Lucas Di Pentima wrote:

Some comments:

• A suggestion for branches with lots of commits: for easier reviewing I think it's better to rebase to main instead of merging main from time to time. Personally I like to look at individual commits to be able to follow the thought process and then do a final pass to the complete diff against main, and it seems it isn't possible when unrelated merge commits are present, because those changes get included too.

Yeah, will do next time sorry.

• At file build/run-build-test-packages-one-target.sh
  • Line 15: the variable $ONLY_BUILD gets replaced with empty string when printing the helpmessage

Indeed! Fixed, also the same bug in `run-build-packages.sh`.

• It's a wrapper for build/run-build-packages-one-target.sh, so should it also accept --arch?

Yeah, why not, I've added that.

• While doing some tests to see how cross-arch package building fails on debian10, it seems that it tries to do it instead of giving the "cannot do it right now" message:

[...]

Ah, right you asked for a specific architecture. I've refactored the tests for (in)valid combinations to also consider that, and to simplify the logic.

• It seems that -j is not necessary, while testing I got the message: "Found user configured '-j' flag in 'rvm_make_flags', please note that RVM can detect number of CPU threads and set the '-j' flag automatically if you do not set it."

It's an RVM bug, RVM's core detection seems to be broken on aarch64, cf. https://github.com/rvm/rvm/issues/4945. That's why I added the -j flag explicitly.

Ready for another look in 4a4e8d3eaa86d08e8fa76d569855247b5131e4bd on branch 17417-add-arm64.

Dev package builds ran successfully at developer-build-packages-multijob: #14

Thanks! this LGTM.