Project

General

Profile

Actions
Copy link

Bug #18732

closed

singularity build permission denied

Added by Peter Amstutz almost 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tom Clegg
Category:
Crunch
Target version:
2022-03-30 Sprint
Start date:
03/18/2022
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release:
Arvados 2.4.0
Release relationship:
Auto

Description

Getting this error:

2022-02-14T15:44:13.011645007Z [singularity build /tmp/crunch-run.x1pr8-dz642-nd5m2icieqol3zo.1865379834/keep2196129160/by_uuid/x1pr8-4zz18-zxxvvxctd0khbp1/image.sif docker-archive:///tmp/crunch-run-singularity-396810067/image.tar]
2022-02-14T15:44:13.035750005Z INFO:    Starting build...
2022-02-14T15:44:13.035750005Z FATAL:   While performing build: conveyor failed to get: error opening file "/tmp/crunch-run-singularity-396810067/image.tar": open /tmp/crunch-run-singularity-396810067/image.tar: permission denied

What seems to be happening is that "singularity build" uses suid to change over to the root user. If crunch-run is launched as a non-root user (and runs arv-mount as non-root), then when singularity changes to root, it can't read from the mount.

The solution is to add "--allow-other" to arv-mount.

Subtasks 3 (0 open — 3 closed)

Task #18778: Investigate behavior on 9tee4ResolvedTom Clegg03/21/2022

Actions
Task #18779: print effective uid and gid in crunch-runResolvedTom Clegg03/20/2022

Actions
Task #18825: Review 18732-crunch-run-log-uidsResolvedWard Vandewege03/18/2022

Actions
Actions
Copy link
 #1

Updated by Peter Amstutz almost 3 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by Peter Amstutz almost 3 years ago

  • Category set to Crunch
  • Assigned To set to Peter Amstutz
Actions
Copy link
 #3

Updated by Peter Amstutz almost 3 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Peter Amstutz almost 3 years ago

  • Target version changed from 2022-02-16 sprint to 2022-03-02 sprint
Actions
Copy link
 #5

Updated by Peter Amstutz almost 3 years ago

  • Target version changed from 2022-03-02 sprint to 2022-03-16 sprint
Actions
Copy link
 #6

Updated by Peter Amstutz almost 3 years ago

  • Release set to 46
Actions
Copy link
 #7

Updated by Peter Amstutz almost 3 years ago

  • Target version changed from 2022-03-16 sprint to 2022-03-30 Sprint
Actions
Copy link
 #8

Updated by Peter Amstutz almost 3 years ago

  • Assigned To changed from Peter Amstutz to Tom Clegg
Actions
Copy link
 #10

Updated by Tom Clegg almost 3 years ago

In #18238, 25bcd259cf59d51263d74ab09dfc4d9a92ca3fa4 adds this:

+    # singularity needs to be owned by root and suid                                                                          
+    chown root /var/lib/arvados/bin/singularity \                                                                             
+          /var/lib/arvados/etc/singularity/singularity.conf \                                                                 
+          /var/lib/arvados/etc/singularity/capability.json \                                                                  
+          /var/lib/arvados/etc/singularity/ecl.toml                                                                           
+    chmod u+s /var/lib/arvados/bin/singularity
This is mysterious because
  • arvados-server install already installs those files with owner=root -- why do they need to be fixed here?
  • This isn't normally how singularity does setuid -- normally .../bin/singularity itself is not setuid, it just invokes .../libexec/singularity/bin/starter-suid (which is setuid) when it needs to elevate privileges -- why does this want to work differently?

IOW, with a normal singularity install, the singularity binary runs as the same user as arv-mount, so allow_other / allow_root isn't needed. But arvbox is doing something different. I expect it would work if we enable allow_root in arvbox, but ideally we would just have arvbox work like a normal singularity install instead.

Actions
Copy link
 #11

Updated by Tom Clegg almost 3 years ago 

    if [[ "$1" != --no-chown ]] ; then
        chown arvbox:arvbox -R /usr/local $ARVADOS_CONTAINER_PATH \
              /var/lib/passenger /var/lib/postgresql \
              /var/lib/nginx /var/log/nginx /etc/ssl/private \
              /var/lib/gopath /var/lib/pip /var/lib/npm \
              /var/lib/arvados
    fi

    mkdir -p /tmp/crunch0 /tmp/crunch1
    chown crunch:crunch -R /tmp/crunch0 /tmp/crunch1

    # singularity needs to be owned by root and suid                                                                           
    chown root /var/lib/arvados/bin/singularity \
          /var/lib/arvados/etc/singularity/singularity.conf \
          /var/lib/arvados/etc/singularity/capability.json \
          /var/lib/arvados/etc/singularity/ecl.toml
    chmod u+s /var/lib/arvados/bin/singularity

Looks like when arvbox's private dir moved from /var/lib/arvados to /var/lib/arvados-arvbox, that chown arvbox:arvbox command added the new dir ($ARVADOS_CONTAINER_PATH is /var/lib/arvados-arvbox) but didn't remove /var/lib/arvados, so it's accidentally sabotaging /var/lib/arvados with chown arvbox:arvbox -R and then partly/incorrectly repairing it with chown root command.

Actions
Copy link
 #13

Updated by Peter Amstutz almost 3 years ago

I removed "/var/lib/arvados" from "chown -R" in arvbox createusers.sh and now it is working, change pushed fe79e3cc0db5662cdb402e1e8de4a82e20368b24

Actions
Copy link
 #14

Updated by Tom Clegg almost 3 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF