Bug #22771: crunch-run handle failure to load image & mark node as broken when out of disk space - Arvados

Actions

Copy link

Bug #22771

closed

crunch-run handle failure to load image & mark node as broken when out of disk space

Added by Peter Amstutz 12 months ago. Updated 11 months ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Tom Clegg

Category:

Crunch

Target version:

Development 2025-04-30

Story points:

Release:

Arvados 3.1.2

Release relationship:

Auto

Description

From #22617

Apr 10 08:49:48 crunch-run[1136]: tordo-dz642-oljtukc4zlwwwpl 2025-04-10T12:49:48.166087313Z loaded image: response {"errorDetail":{"message":"write /usr/local/lib/R/site-library/BH/include/boost/archive/iterators/xml_escape.hpp: no space left on device"},"error":"write /usr/local/lib/R/site-library/BH/include/boost/archive/iterators/xml_escape.hpp: no space left on device"}

We get a response with "errorDetail" but the Docker SDK doesn't set the "err" return value. We should Unmarshal the response and see if "error" is nonempty and also return that as an error.
The substring "no space left on device" should be added to the broken node blacklist

Subtasks 1 (0 open — 1 closed)

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Peter Amstutz 12 months ago

Position changed from -950389 to -950377
Status changed from New to In Progress

Actions

Copy link

Updated by Peter Amstutz 12 months ago

Status changed from In Progress to New
Category set to Crunch
Subject changed from Handle failure to load image & mark node as broken when out of disk space to crunch-run handle failure to load image & mark node as broken when out of disk space

Actions

Copy link

Updated by Peter Amstutz 12 months ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz 12 months ago

Target version changed from Development 2025-04-16 to Development 2025-04-30

Actions

Copy link

Updated by Tom Clegg 11 months ago

Assigned To set to Tom Clegg

Actions

Copy link

Updated by Tom Clegg 11 months ago

Subtask #22796 added

Actions

Copy link

Updated by Tom Clegg 11 months ago

22617-docker-load-error @ 28e67f3dc610b8882c24a1060c8983f2fe3cd25e -- developer-run-tests: #4755

All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
- ✅ Propagate errors from the docker daemon response
- ✅ Add "no space left on device" to list of phrases that indicate a broken node
- ✨ Improve some repetitive error messages ("while loading image: while loading image: ...")
Anything not implemented (discovered or discussed during work) has a follow-up story.
- n/a
Code is tested and passing, both automated and manual, what manual testing was done is described.
- ✅ Add test for error propagation
New or changed UX/UX and has gotten feedback from stakeholders.
- n/a
Documentation has been updated.
- n/a
Behaves appropriately at the intended scale (describe intended scale).
- n/a
Considered backwards and forwards compatibility issues between client and server.
- n/a
Follows our coding standards and GUI style guidelines.
- ✅

Actions

Copy link

Updated by Peter Amstutz 11 months ago

LGTM, although there's a separate recurring issue with the tests failing "ValueError: cannot mmap an empty file", but this isn't the only branch affected by that.

Actions

Copy link