Bug #15721: OPTIONS on /arvados/v1/config returns status 503 - Arvados

Actions

Copy link

Bug #15721

closed

OPTIONS on /arvados/v1/config returns status 503

Added by Peter Amstutz about 5 years ago. Updated almost 5 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Tom Clegg

Category:

Target version:

2019-10-23 Sprint

Start date:

10/15/2019

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Story points:

Release:

Arvados 2.0 - Early 2020

Release relationship:

Auto

Description

Requesting OPTIONS on /arvados/v1/config returns a 503 status code:

$ curl -v -X OPTIONS https://c97qk.arvadosapi.com/arvados/v1/config
...
< HTTP/1.1 503 Service Unavailable
< Server: nginx/1.15.8
< Date: Fri, 11 Oct 2019 20:22:24 GMT
< Content-Length: 0
< Connection: keep-alive
< Access-Control-Allow-Headers: Authorization, Content-Type
< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 86486400
< 
* Curl_http_done: called premature == 0
* Connection #0 to host c97qk.arvadosapi.com left intact

This is a bug in master, since it works on 1.4.1:

$ curl -v -X OPTIONS https://qr1hi.arvadosapi.com/arvados/v1/config
...
< HTTP/1.1 200 OK
< Server: nginx/1.15.8
< Date: Fri, 11 Oct 2019 20:23:45 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Access-Control-Allow-Headers: Authorization, Content-Type
< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 86486400
< Cache-Control: no-cache
< Status: 200 OK
< Strict-Transport-Security: max-age=15552000
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Powered-By: Phusion Passenger 6.0.2
< X-Request-Id: req-15o843ywr0rr16tit4ws
< X-Runtime: 0.003235
< X-Xss-Protection: 1; mode=block
< 
* Curl_http_done: called premature == 0
* Connection #0 to host qr1hi.arvadosapi.com left intact

This is a blocker because it breaks workbench 2 multi-site search (found while working on #15530)

Subtasks 1 (0 open — 1 closed)

Actions

Copy link

Updated by Peter Amstutz about 5 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Peter Amstutz about 5 years ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz about 5 years ago

Status changed from In Progress to New

Actions

Copy link

Updated by Peter Amstutz about 5 years ago

Description updated (diff)
Priority changed from Normal to High

Actions

Copy link

Updated by Tom Clegg about 5 years ago

Hm. I wouldn't expect OPTIONS to return 503 -- but I wouldn't expect an OPTIONS (preflight) request for the config endpoint at all, since the actual request is a GET...

Actions

Copy link

Updated by Tom Clegg about 5 years ago

Assigned To set to Tom Clegg

Actions

Copy link

Updated by Peter Amstutz about 5 years ago

Tom Clegg wrote:

Hm. I wouldn't expect OPTIONS to return 503 -- but I wouldn't expect an OPTIONS (preflight) request for the config endpoint at all, since the actual request is a GET...

It might be sending an authorization header or something that forces a preflight request. I didn't look too closely since this is something that used to work, I don't think workbench2 is doing anything wrong.

Actions

Copy link

Updated by Tom Clegg about 5 years ago

Missing "return" statement. In the "no request limit configured" case we were calling the handler, then returning 503 because we'd already reached the maximum of 0 concurrent requests. Same bug was also causing a lot of this:

2019/10/15 14:10:18 http: superfluous response.WriteHeader call from git.curoverse.com/arvados.git/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)

15721-config-cors @ ef25437d74cd7c2ce9ba4df2364c1da3d06b8bbb -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1585/

If you need a workaround -- configure MaxConcurrentRequests: 1000

Actions

Copy link

Updated by Tom Clegg about 5 years ago

Status changed from New to In Progress

Actions

Copy link

#10

Updated by Peter Amstutz about 5 years ago

Tom Clegg wrote:

Missing "return" statement. In the "no request limit configured" case we were calling the handler, then returning 503 because we'd already reached the maximum of 0 concurrent requests. Same bug was also causing a lot of this:

2019/10/15 14:10:18 http: superfluous response.WriteHeader call from git.curoverse.com/arvados.git/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)

15721-config-cors @ ef25437d74cd7c2ce9ba4df2364c1da3d06b8bbb -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1585/

LGTM

If you need a workaround -- configure MaxConcurrentRequests: 1000

We could also set a default value for MaxConcurrentRequests and require that it be > 0, seems like one of those things where you always want a limit to mitigate accidental or deliberate denial of service.

Actions

Copy link

#11

Updated by Tom Clegg about 5 years ago

Changing how we spell "unlimited" from "0" to "99999999" (for the sake of reserving "0" to mean "refuse to start") doesn't sound appealing to me. I don't object to changing the default although it doesn't feel like a priority.

Actions

Copy link

#12