Arvados

I haven't been able to reproduce the problem using a local virtualenv and PySDK at master.

Nevertheless, I've found a pycurl bug report that seem to be similar to this issue: https://github.com/pycurl/pycurl/issues/413

The odd thing is that the code listed to reproduce the problem seem to work (that is, fails as reported) on my dev environment.

This sounds not just similar, but identical, to me (including the fact that it occurs with HEADERFUNCTION): https://github.com/pycurl/pycurl/issues/413#issuecomment-276954718

The workaround: https://github.com/lorien/grab/commit/f26fd3052aaa9d9b24f1098759d73dcc2c00d7fa#diff-50f780fd1dcf1fa66656abf741a55c36L489

seems a little convoluted to me, so I'm not 100% sure we want to go down that path, but perhaps you could discuss with Tom and/or Peter to brainstorm ideas for a possible solution.

It sounds like we need to do something like this in order to detect ^C properly.

I've seen this issue with my own program that uses the Python SDK and doesn't do any exception catching, so perhaps the fix is needed in the Python SDK, not in arv-get.

This took three successive ctrl/Cs to kill:

j VGX2048 callvars e51c5-8i9sb-825qzt1ytjua375 2/2 cores (req/actual), 10000/14020 MB, 1.77 hours, 3.55 core-hours, $ 0.26
^CTraceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/arvados/keep.py", line 524, in _headerfunction
    def _headerfunction(self, header_line):
KeyboardInterrupt
j VGX2048 callvars e51c5-8i9sb-3g1m40e6c3a9sbm 2/2 cores (req/actual), 10000/14011 MB, 1.53 hours, 3.06 core-hours, $ 0.23
j VGX2048 callvars e51c5-8i9sb-lpjhzrsg73fvt4c 2/2 cores (req/actual), 10000/14011 MB, 2.17 hours, 4.34 core-hours, $ 0.32
...
j VGX2048 scatter-samtools-view-sort-index e51c5-8i9sb-iunc8f5dhh9vu9b 2/2 cores (req/actual), 10000/14011 MB, 0.23 hours, 0.46 core-hours, $ 0.03
j VGX2048 scatter-samtools-view-sort-index e51c5-8i9sb-uu0zd3buzscnm0k 2/20 cores (req/actual), 10000/141017 MB, 0.26 hours, 5.10 core-hours, $ 0.38
j VGX2048 scatter-samtools-view-sort-index e51c5-8i9sb-00w6k80myf8cwym 2/16 cores (req/actual), 10000/56347 MB, 0.35 hours, 5.60 core-hours, $ 0.32
^CTraceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/arvados/keep.py", line 367, in <lambda>
    lambda *args, **kwargs: self._socket_open(*args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/arvados/keep.py", line 335, in _socket_open
    return self._socket_open_pycurl_7_19_3(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/arvados/keep.py", line 341, in _socket_open_pycurl_7_19_3
    'Address', ['family', 'socktype', 'protocol', 'addr'],
  File "/usr/lib/python2.7/collections.py", line 386, in namedtuple
    exec class_definition in namespace
  File "<string>", line 1, in <module>
KeyboardInterrupt
j VGX2048 scatter-samtools-view-sort-index e51c5-8i9sb-fmhodqsywh89uss 2/2 cores (req/actual), 10000/14020 MB, 0.29 hours, 0.57 core-hours, $ 0.04