Arvados

Looks like this can be done with "nsenter" if you're root or add some capabilities (including the dreaded cap_sys_admin) and know the PID of a process in the container:

$ echo $$
675765
$ singularity exec --containall --cleanenv docker://debian:11 /bin/bash
Singularity> df /
Filesystem     1K-blocks  Used Available Use% Mounted on
overlay            16384    16     16368   1% /
Singularity> 

$ cp -p /usr/bin/nsenter /tmp/nsenter
$ sudo setcap "cap_sys_admin+pei cap_sys_chroot+pei" /tmp/nsenter
$ pstree -up 675765
bash(675765,tom)───starter-suid(838413)─┬─sinit(838434)─┬─bash(838456)
...
$ /tmp/nsenter --target 838456 --all df /
Filesystem     1K-blocks  Used Available Use% Mounted on
overlay            16384    16     16368   1% /

Surely there is a reasonable way to find the pid of the contained process.

Yes, I think the target can be any process inside the container, doesn't need to be the topmost one. But it does need to be the right container even when #14922 happens, other singularity processes are running for other reasons, etc. We could run lsns and look for a pid whose Nth great-grandparent matches the PID in the crunch-run lockfile...?

Tom Clegg wrote:

Yes, I think the target can be any process inside the container, doesn't need to be the topmost one. But it does need to be the right container even when #14922 happens, other singularity processes are running for other reasons, etc. We could run lsns and look for a pid whose Nth great-grandparent matches the PID in the crunch-run lockfile...?

Yes, since we know the pid of the process we started, we should be able to figure out the container pretty easily, since it's just a regular process tree without the shenanagins that Docker does.