Arvados

• Enhance crunchstat so that available free space is periodically logged along with the already present mem, cpu & i/o stats.
• To avoid having to be calling the df command, golang provides a syscall package with an appropriate Statfs function.
• node-info logs already record the available space and i-nodes at the start, so this addition would complement that information.

Should log all three figures (available, used, total). Generally available+used<total because the "available only to root" portion is not counted as available. Reporting tools can decide the most useful way to report.

Updates at e4c31590b - branch 12444-tmpdir-monitoring
Test run: https://ci.curoverse.com/job/developer-run-tests/821/

Adds tmpdir stats to crunchstat reporting: available, used & total with usage increments.

There's a sdk/python test failure, looking into it.

As suggested by Peter, rebasing on latest master fixed the issue.

Now at b211e857d304f7fbe8787d2b65a307da841d047b
Test run: https://ci.curoverse.com/job/developer-run-tests-sdk-python-ruby/108/

    err := syscall.Statfs("/tmp", &s)

Shouldn't be hardcoded. By default it should use $TMPDIR, but it would be better to pass it in. Crunch-run should be updated to pass in runner.parentTemp.

        total:      s.Blocks * bs,
        used:       (s.Blocks - s.Bavail) * bs,
        available:  s.Bavail * bs,

Should be used: (s.Blocks - s.Bfree) * bs

(because Bavail < Bfree)

    r.Logger.Printf("tmpdir available:%d used:%d total:%d%s\n",
        nextSample.available, nextSample.used, nextSample.total, delta)

I don't think this formatting is consistent with the other crunchstat lines (which have the number first, then the name), should be something like:

    r.Logger.Printf("statfs %d available %d used %d total%s\n", 
        nextSample.available, nextSample.used, nextSample.total, delta)

Updates at 85f6919fa
Test run: https://ci.curoverse.com/job/developer-run-tests/824/

Addressed above comments and added a notice on the log to report which directory is being monitored.

Lucas Di Pentima wrote:

Updates at 85f6919fa
Test run: https://ci.curoverse.com/job/developer-run-tests/824/

Addressed above comments and added a notice on the log to report which directory is being monitored.

This LGTM.

Could you file a follow-on ticket to update crunchstat-summary to graph disk usage?