Project

General

Profile

Actions

Bug #9001

closed

crunchstat-summary fails on year old log file

Added by Bryan Cosca almost 10 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Peter Amstutz
Category:
-
Target version:
-
Story points:
-

Description

Trying to use it on a job from 2/24/2015. I guess something changed with the way we log things and it hits a ValueError.

~/arvados/tools/crunchstat-summary/bin/crunchstat-summary --job qr1hi-8i9sb-030iriqh13rvfok --format text > qr1hi-8i9sb-030iriqh13rvfok.txt
Traceback (most recent call last):
  File "/home/bcosc/arvados/tools/crunchstat-summary/bin/crunchstat-summary", line 14, in <module>
    cmd.run()
  File "/home/bcosc/.local/lib/python2.7/site-packages/crunchstat_summary-0.1.20160412153631-py2.7.egg/crunchstat_summary/command.py", line 56, in run
    return self.summer.run()
  File "/home/bcosc/.local/lib/python2.7/site-packages/crunchstat_summary-0.1.20160412153631-py2.7.egg/crunchstat_summary/summarizer.py", line 150, in run
    stat, line, e))
ValueError: Error parsing run stat in "2015-02-24_16:52:10 qr1hi-8i9sb-030iriqh13rvfok 8083 0 stderr crunchstat: Running [/usr/bin/docker.io run --rm=true --attach=stdout --attach=stderr --attach=stdin -i --user=crunch --cidfile=/tmp/crunch-job/qr1hi-ot0gb-pqig664l3n38kvz.cid --sig-proxy --dns 10.13.66.165 --dns 172.17.42.1 --volume=/tmp/crunch-job/src:/tmp/crunch-job/src:ro --volume=/tmp/crunch-job/opt:/tmp/crunch-job/opt:ro --volume=/tmp/crunch-job/task/compute38.1.keep:/keep:ro --volume=/tmp --env=CRUNCH_JOB_BIN=/usr/local/arvados/src/services/crunch/crunch-job --env=TASK_SEQUENCE=0 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_COMMAND=ARRAY(0x2cc7c80) --env=CRUNCH_SRC_COMMIT=83a9390a05bbffc2e4ea95dd693af3ab3547fa12 --env=TASK_QSEQUENCE=0 --env=JOB_PARAMETER_REFERENCE=3514b8e5da0e8d109946bc809b20a78a+5698 --env=CRUNCH_INSTALL=/tmp/crunch-job/opt --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=4c3mmeij7flwquw30x6mqbejvrooyfeiq3ccvmb1mwjad1nzrk --env=CRUNCH_WORK=/tmp/crunch-job/work --env=CRUNCH_TMP=/tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/compute38.1 --env=JOB_UUID=qr1hi-8i9sb-030iriqh13rvfok --env=JOB_PARAMETER_TASK.FOREACH=ARRAY(0x2cb33d0) --env=JOB_PARAMETER_B=HASH(0x2a9dbf0) --env=CRUNCH_JOB_UUID=qr1hi-8i9sb-030iriqh13rvfok --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=compute38 --env=JOB_SCRIPT=run-command --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_SAMPLE_ID=my_sample --env=JOB_PARAMETER_TASK.STDOUT=$(param) --env=JOB_PARAMETER_D=d0136bc494c21f79fc1b6a390561e6cb+2778 --env=JOB_PARAMETER_PARAM=HASH(0x2cc7cb0) --env=TASK_WORK=/tmp/crunch-job-task-work/compute38.1 --env=ARVADOS_API_HOST=qr1hi.arvadosapi.com --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=qr1hi-ot0gb-pqig664l3n38kvz --env=JOB_PARAMETER_EACHREAD=$(dir $(d)) --env=CRUNCH_SRC=/tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/compute38.1 f4eafaf1e2d738e0f8d947feb725b5945f0219c5c4956eec6e164a0788abbab8 stdbuf --output=0 --error=0 perl - /tmp/crunch-job/src/crunch_scripts/run-command]
": ValueError('could not convert string to float: [/usr/bin/docker.io',)
Actions #1

Updated by Brett Smith almost 10 years ago

Bryan,

Your diagnosis is spot-on: crunchstat-summary doesn't know how to parse too-old logs. How often do you wish it could?

Actions #2

Updated by Bryan Cosca almost 10 years ago

I think not often enough to change the code (it's only happened once)

Actions #3

Updated by Brett Smith almost 10 years ago

  • Target version set to Arvados Future Sprints
Actions #4

Updated by Peter Amstutz over 9 years ago

Branch @ 9001-summary-skip-malformed-lines

Actions #5

Updated by Lucas Di Pentima over 9 years ago

Reviewed the fix, looks good to me. Please merge.

Actions #6

Updated by Peter Amstutz over 9 years ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:6c0bf267d795a3ca49c3258c9490714c9e18d333.

Actions #7

Updated by Tom Morris about 9 years ago

  • Assigned To set to Peter Amstutz
  • Target version deleted (Arvados Future Sprints)
Actions

Also available in: Atom PDF