Arvados

TestInject in source:lib/crunchrun/singularity_test.go looks like this:

func (s *singularitySuite) TestInject(c *C) {
    path, err := exec.LookPath("nsenter")
    if err != nil || path != "/var/lib/arvados/bin/nsenter" {
        c.Skip("looks like /var/lib/arvados/bin/nsenter is not installed -- re-run `arvados-server install`?")
    }
    s.executorSuite.TestInject(c)
}

As we've gotten away from arvados-server install, this test basically doesn't run. It's not running on Jenkins as of October 2025. And if you just remove the path condition, the test fails like this:

----------------------------------------------------------------------
FAIL: singularity_test.go:62: singularitySuite.TestInject

building singularity image
[singularity build /tmp/crunch-run-singularity-3405259366/image.sif docker-archive:///tmp/crunch-run-singularity-3405259366/image.tar]
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:67f770da229bf16d0c280f232629b0c1f1243a884df09f6b940a1c7288535a6d
Copying config sha256:a11e762410a6fb4e925d1ea535fecc177d983bdf0dba3261d244fb3c7ee18865
Writing manifest to image destination
Storing signatures
2025/10/15 21:24:08  info unpack layer: sha256:378e3b9fb50c743e1daa7a79dc2cf7c18aa0ac8137a1ca0d51a3b909c80e7d48
INFO:    Creating SIF file...
INFO:    Build complete: /tmp/crunch-run-singularity-3405259366/image.sif

inject [cat /injecttest-33196/testfile] => "nsenter: reassociate to namespace 'ns/cgroup' failed: Operation not permitted\n" 
singularity_test.go:67:
    s.executorSuite.TestInject(c)
executor_test.go:289:
    c.Check(err, IsNil)
... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc0028c0540), Stderr:[]uint8(nil)} ("exit status 1")

singularity_test.go:67:
    s.executorSuite.TestInject(c)
executor_test.go:290:
    c.Check(string(out), Equals, "first tube")
... obtained string = "nsenter: reassociate to namespace 'ns/cgroup' failed: Operation not permitted\n" 
... expected string = "first tube" 

container ran for 268.882353ms
OOPS: 118 passed, 2 skipped, 1 FAILED
--- FAIL: TestCrunchExec (96.86s)

I'm guessing arvados-server install either built Singularity with particular options or configured the system in some way to allow this to work. What was that way? Should we do the same thing on Jenkins? Is there another way we can implement this test?