Project

General

Profile

Actions
Copy link

Bug #18293

closed

MemoryError creating SIF

Added by Peter Amstutz about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
2021-10-27 sprint
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-
Release:
Arvados 2.3.0
Release relationship:
Auto

Description

https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-tzmwfcbx9r415ed#Log

crunch-run.txt:

While performing build: while creating SIF: while creating container: writing data object for SIF file: copying data object file to SIF file: write /tmp/crunch-run.9tee4-dz642-0prt4gvmv3qkmna.1720801655/keep2342085199/by_uuid/9tee4-4zz18-i232kkojjjrtrt0/image.sif: input/output error

arv-mount.txt:

2021-10-25T20:46:57.239059516Z 2021-10-25 20:46:57 arvados.arvados_fuse3532322 ERROR: Unhandled exception during FUSE operation
2021-10-25T20:46:57.239059516Z Traceback (most recent call last):
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados_fuse/__init__.py", line 327, in catch_exceptions_wrapper
2021-10-25T20:46:57.239059516Z return orig_func(self, *args, **kwargs)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados_fuse/__init__.py", line 676, in write
2021-10-25T20:46:57.239059516Z w = handle.obj.writeto(off, buf, self.num_retries)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados_fuse/fusefile.py", line 69, in writeto
2021-10-25T20:46:57.239059516Z return self.arvfile.writeto(off, buf, num_retries)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 469, in must_be_writable_wrapper
2021-10-25T20:46:57.239059516Z return orig_func(self, *args, **kwargs)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
2021-10-25T20:46:57.239059516Z return orig_func(self, *args, **kwargs)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 1153, in writeto
2021-10-25T20:46:57.239059516Z self._current_bblock.repack_writes()
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
2021-10-25T20:46:57.239059516Z return orig_func(self, *args, **kwargs)
2021-10-25T20:46:57.239059516Z File "/usr/share/python3/dist/python3-arvados-fuse/lib/python3.7/site-packages/arvados/arvfile.py", line 436, in repack_writes
2021-10-25T20:46:57.239059516Z new_bb.append(contents[t.segment_offset:t.segment_offset+t.range_size])
2021-10-25T20:46:57.239059516Z MemoryError

Actions
Copy link
 #1

Updated by Peter Amstutz about 3 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by Peter Amstutz about 3 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Peter Amstutz about 3 years ago

  • Release set to 42
Actions
Copy link
 #4

Updated by Ward Vandewege about 3 years ago

Both compute nodes on 9tee4 have

mksquashfs mem = 512M

in their /etc/singularity/singularity.conf file.

Actions
Copy link
 #5

Updated by Peter Amstutz about 3 years ago

  • Status changed from In Progress to Resolved

We've lowered the recommended memory limit for mksquashfs, and also confirmed that we can work around it by increasing RequestExtraRAM to the config.

Actions
Copy link

Also available in: Atom PDF