Arvados

arvados-dev branch 9309-centos7-uplaods-wip is branch #2, and super-short.

reviewing branch 9309-centos7-uplaods-wip on arvados-dev: LGTM.

reviewing 9309-centos-7-packages-wip:

• this is going to need to be rebased to master; there are some changes in build/run-build-packages.sh that aren't reflected yet here (the second hunk on build/run-build-packages.sh won't apply)

• line 258 in run-library.sh, where there's a refactoring of fpm_dirs to fpm_dir:

- it seems to me that the test for $PACKAGE_NAME being defined is true in the example given for llfuse=0.41.1. Does that mean the 'else' condition is superfluous?

Otherwise, LGTM: I built all the centos7 packages, and that worked. The tests didn't pass but then this is a 'wip' branch, so!

Ward Vandewege wrote:

reviewing 9309-centos-7-packages-wip:

• this is going to need to be rebased to master; there are some changes in build/run-build-packages.sh that aren't reflected yet here (the second hunk on build/run-build-packages.sh won't apply)

Done, although there were no merge conflicts in the process...

• line 258 in run-library.sh, where there's a refactoring of fpm_dirs to fpm_dir:

- it seems to me that the test for $PACKAGE_NAME being defined is true in the example given for llfuse=0.41.1. Does that mean the 'else' condition is superfluous?

Ugh, no, it means I introduced a bug. I didn't realize we were setting $PACKAGE_NAME in that case. Things happened to work on distros where $PYTHON2_PKG_PREFIX == $PACKAGE_TYPE, but I broke fpm-info.sh on other distros like centos6. I have to add some new scheme here in order to accommodate libcloud, where we can't add fpm-info.sh to the source directory (because it's fundamentally not our repo) and $PACKAGE is not predictable (because it's a random tempdir).

I have reverted to having code closer to master, that adds backports/$PACKAGE_NAME as another place to search.

Otherwise, LGTM: I built all the centos7 packages, and that worked. The tests didn't pass but then this is a 'wip' branch, so!

The tests failed because of the CWL package-related brokenness that we fixed in master over the weekend. Now that we're rebased on master, things should be good. Now at 35abd53.

Brett Smith wrote:

Ward Vandewege wrote:

reviewing 9309-centos-7-packages-wip:

• this is going to need to be rebased to master; there are some changes in build/run-build-packages.sh that aren't reflected yet here (the second hunk on build/run-build-packages.sh won't apply)

Done, although there were no merge conflicts in the process...

Great.

• line 258 in run-library.sh, where there's a refactoring of fpm_dirs to fpm_dir:

- it seems to me that the test for $PACKAGE_NAME being defined is true in the example given for llfuse=0.41.1. Does that mean the 'else' condition is superfluous?

Ugh, no, it means I introduced a bug. I didn't realize we were setting $PACKAGE_NAME in that case. Things happened to work on distros where $PYTHON2_PKG_PREFIX == $PACKAGE_TYPE, but I broke fpm-info.sh on other distros like centos6. I have to add some new scheme here in order to accommodate libcloud, where we can't add fpm-info.sh to the source directory (because it's fundamentally not our repo) and $PACKAGE is not predictable (because it's a random tempdir).

Yeah, ok.

I have reverted to having code closer to master, that adds backports/$PACKAGE_NAME as another place to search.

Otherwise, LGTM: I built all the centos7 packages, and that worked. The tests didn't pass but then this is a 'wip' branch, so!

The tests failed because of the CWL package-related brokenness that we fixed in master over the weekend. Now that we're rebased on master, things should be good. Now at 35abd53.

That code looks good. I see a package build failure now, which was invisible due to backports/python-llfuse/fpm-info.sh not being pulled in before:

Loading fpm overrides from /arvados/backports/python-llfuse/fpm-info.sh
ls: cannot access /arvados/packages/centos7/fuse-devel-[0-9]*.rpm: No such file or directory
Attemping to install build_dep fuse-devel
error: open of fuse-devel failed: No such file or directory

Ward Vandewege wrote:

That code looks good. I see a package build failure now, which was invisible due to backports/python-llfuse/fpm-info.sh not being pulled in before:

That's not a build failure: check your package directory, and you should have python-llfuse there. These errors come from the part of fpm_build where we try to install build_depends. ls fails because we didn't build fuse-devel locally, and then rpm -i fuse-devel fails because that doesn't even make sense (it should probably say yum install -q -y fuse-devel instead). But the Docker image already has fuse-devel installed, so the main build still succeeds.

In my own run, building the keep-balance package failed... I'm looking into that now.

Brett Smith wrote:

Ward Vandewege wrote:

That code looks good. I see a package build failure now, which was invisible due to backports/python-llfuse/fpm-info.sh not being pulled in before:

That's not a build failure: check your package directory, and you should have python-llfuse there. These errors come from the part of fpm_build where we try to install build_depends. ls fails because we didn't build fuse-devel locally, and then rpm -i fuse-devel fails because that doesn't even make sense (it should probably say yum install -q -y fuse-devel instead). But the Docker image already has fuse-devel installed, so the main build still succeeds.

In my own run, building the keep-balance package failed... I'm looking into that now.

You are right. The build failed on keep-balance for me too, the thing about python-llfuse was not a build failure. I didn't scroll up enough.

It looks like the keep-balance failure is just that the Dockerfile needs to be updated to reflect the new name of the Go tarball. I'm trying that now.

LGTM now, thanks!

Build process for the extracirricular packages was mostly the same as before in #8005-12, with just minor variations:

<https://codeload.github.com/dun/munge/legacy.tar.gz/880584f51f1a4b3641552bfae9f89e1f1c9a2fda>
    yum install bzip2-devel
    tar -xf munge-0.5.12.tar.gz
    mv dun-munge-*/ munge-0.5.12
    tar -c munge-0.5.12 | xz > munge-0.5.12.tar.xz
    rpmbuild -tb munge-0.5.12.tar.xz
    rpmbuild -ts munge-0.5.12.tar.xz

<http://www.schedmd.com/download/latest/slurm-16.05.0.tar.bz2>
     yum install pam-devel openssl
     rpm -ivh ~/rpmbuild/RPMS/x86_64/munge*.rpm
     rpmbuild -tb slurm-16.05.0.tar.bz2
     rpmbuild -ts slurm-16.05.0.tar.bz2

## runit
     yum -y install rpmdevtools git glibc-static
     git clone https://github.com/imeyer/runit-rpm runit-rpm
     cd runit-rpm/
     git checkout eefeaca0913638f375afe92d03c5b4c5f81128f2
     ./build.sh
     rpmbuild -bs ~/rpmbuild/SPECS/runit.spec

New package build is failing. This seems to be why.

The most commonly recommended workaround in that thread is to use the devicemapper storage backend.

Didn't happen to me on my workstation, running Debian 8 with a Linux 4.3 backport and just a stock Docker configuration.

My workstation is using the devicemapper driver (even though this isn't specified in any config that I've found yet), while ci is using aufs, so that's fundamentally the distinction causing different results.

(10:17:27 AM) Me: Do people have strong opinions about adding/splitting off an Install Guide page, before the SSO install page, titled something like "Set up PostgreSQL database(s)"?
(10:17:53 AM) Me: Deployers have a lot of flexibility in the database setup, so splitting that off gives us a little space to cover the options and let people pick what they want at the right time.
(10:18:17 AM) Me: The database setup is also one of the more distro-specific parts of the Install Guide, so splitting it off can help make other pages less fork-y.
(10:19:44 AM) ward: sounds good to me!
(10:20:46 AM) tetron_: agree

9309-postgresql-install-guide-wip is up for review to make the change discussed above. It is pure refactoring; it does not add anything about CentOS 7.

9309-centos-7-install-guide-wip is up for review. I didn't follow literally the entire install guide, but I did go through all the steps that involved installing any of our packages, and setting up stuff around that. I also installed third-party software like Gitolite. But I didn't step through creating API objects and so on; I figure that's high-level enough that it's low-risk.

The weirdest thing is the process for installing the FUSE package. There's a python2-llfuse package in epel that provides llfuse 1.01 or thereabouts, and it obsoletes the python-llfuse package. However, later versions of llfuse have breaking changes, so we really need to have the exact version we've backported. Telling yum to ignore the repository in the meantime seemed like the easiest way forward.

If you want to test things out, you will need to check out 9309-centos-7-rails-scripts-wip and build packages from it. Follow this blog post to create and start a CentOS 7 Docker image running systemd.

The branch probably shouldn't be merged until the rails-scripts branch is merged and packages are actually built (which requires some Jenkins work too).

Brett Smith wrote:

The weirdest thing is the process for installing the FUSE package. There's a python2-llfuse package in epel that provides llfuse 1.01 or thereabouts, and it obsoletes the python-llfuse package. However, later versions of llfuse have breaking changes, so we really need to have the exact version we've backported. Telling yum to ignore the repository in the meantime seemed like the easiest way forward.

The specific problem:

--> Processing Dependency: python-llfuse = 0.41.1 for package: python-arvados-fuse-0.1.20160426220707-2.noarch
Package python-llfuse is obsoleted by python2-llfuse, but obsoleting package does not provide for requirements
Error: Package: python-arvados-fuse-0.1.20160426220707-2.noarch (localrepo)
           Requires: python-llfuse = 0.41.1
           Available: python-llfuse-0.41.1-3.x86_64 (localrepo)
               python-llfuse = 0.41.1-3
           Available: python2-llfuse-1.0-1.el7.x86_64 (epel)
               python-llfuse = 1.0-1.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

test df8d16f045c8ed9d5fa7de35c05ab4d4fe234853

I'm following the instructions and we use sudo a lot, expecting the user to be the one excecuting the instructions. but sometimes it back fires:

[root@centos-2gb-nyc2-01 ruby-2.1.7]# sudo gem install bundler
sudo: gem: command not found
[root@centos-2gb-nyc2-01 ruby-2.1.7]# gem install bundler
Fetching: bundler-1.12.5.gem (100%)

this is from install-sso.html under "Build and install Ruby" on CentOS 7.2

we should use sudo -i gem ... instead.

But this is ready to merge event with that issue in place

Nico Cesar wrote:

this is from install-sso.html under "Build and install Ruby" on CentOS 7.2

we should use sudo -i gem ... instead.

This is genuinely surprising. If you're using option #2 to build and install Ruby--so you're building from source, and not using RVM--I would expect gem to be somewhere under $PATH after sudo make install. Where did it go?

Brett Smith wrote:

This is genuinely surprising. If you're using option #2 to build and install Ruby--so you're building from source, and not using RVM--I would expect gem to be somewhere under $PATH after sudo make install. Where did it go?

I see: on Red Hat, the /usr/local paths are added to $PATH in /etc/profile. OK, added sudo -i to gem install lines in 2bf563e.

test 61284297350777ec0e752f272c0bd6ffbde81af7 / 9309-centos-7-rails-scripts-wip

I notice that #!/bin/sh was used on common-test-rails-server-package.sh then later some bashisms like ${PACKAGE_NAME%-server} and ${PACKAGE_NAME%.sh} .. probably the safest is to explicitly use a /bin/bash she-bang. but otherwise I find the script to be ok. (probably I'm too picky with this, I had some problems in the past with weird setups )

Besides that, the re-organization looked good to me.

I haven't done a try all the combinatios of packages and distros for build/rails-package-scripts/postinst.sh which I assume we'll be iterating over time to see if this works everywhere in present and future configurations.

Ready to merge-

Nico Cesar wrote:

I notice that #!/bin/sh was used on common-test-rails-server-package.sh then later some bashisms like ${PACKAGE_NAME%-server} and ${PACKAGE_NAME%.sh}

?? Those are not bashisms. man dash, search for "Parameter Expansion"

I haven't done a try all the combinatios of packages and distros for build/rails-package-scripts/postinst.sh which I assume we'll be iterating over time to see if this works everywhere in present and future configurations.

FWIW the package tests do go through the postinst, and I ran them for centos6, centos7, and debian8 without issues.

review arvados-dev @ commit:101791be5e0f127a46971d2c2b72d45e4f9e4fef

this is Ready to merge.

I'll do the deploy in CI so

https://ci.curoverse.com/view/Arvados%20build%20pipeline/job/build-packages-python-ruby/

goes back to blue