Arvados

22791-s3-upload-concurrency @ 68a4d6123b466daad93c8027a19f3de981df4439 -- developer-run-tests: #4776

22791-s3-upload-concurrency @ f3a8012a3f7e81e891ab6ba9cbb734bd49c632bb -- developer-run-tests: #4780

Failure seems unrelated:

FAIL: singularity_test.go:245: singularitySuite.TestImageCache_Concurrency_10
...
INFO:    Creating SIF file...
FATAL:   While performing build: while creating SIF: while creating container: open /tmp/check-1596490754/26/by_uuid/zzzzz-4zz18-mkdap9nefs0fjr6/image.sif: no such file or directory

singularity_test.go:270:
    c.Check(err, IsNil)
... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc004097158), Stderr:[]uint8(nil)} ("exit status 255")

(this might be a real unrelated issue, although I haven't been able to reproduce it locally yet)

22791-s3-upload-concurrency @ f3a8012a3f7e81e891ab6ba9cbb734bd49c632bb -- developer-run-tests: #4781

• All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
  • ✅ file uploads via keep-web's s3 interface are race-safe
  • ✨ file deletions via keep-web's s3 interface are race-safe
  • Approach is the same as the WebDAV code path: use replace_files for uploading (instead of writing them to the cached filesystem), and subsequent downloads use needSync() to detect whether the cached filesystem needs to be refreshed
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • n/a
• Code is tested and passing, both automated and manual, what manual testing was done is described.
  • ✅ Added a test for concurrent uploads via S3
• New or changed UX/UX and has gotten feedback from stakeholders.
  • n/a
• Documentation has been updated.
  • n/a
• Behaves appropriately at the intended scale (describe intended scale).
  • ✅ Performance should be the same as for WebDAV uploads
• Considered backwards and forwards compatibility issues between client and server.
  • ✅ No concerns, the previous version of keep-web already depended on the replace_files API for WebDAV uploads
• Follows our coding standards and GUI style guidelines.
  • ✅

Tom Clegg wrote in #note-8:

22791-s3-upload-concurrency @ f3a8012a3f7e81e891ab6ba9cbb734bd49c632bb -- developer-run-tests: #4781

LGTM, thanks.