Actions
Story #16222
openHandle container live logs in a more scalable way
Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:
0%
Estimated time:
Story points:
-
Release:
Release relationship:
Auto
Description
Containers send their live logs to the database. Unfortunately when there are a very large number of containers, this can overwhelm the database and cause the API server to become non-responsive and start returning 503 errors.
We need better system behaviors and/or a new architecture so that large logging volumes do not cripple the system, and ideally don't require extensive tuning like the current logging parameters do, which only ever happens after a critical failure.
This solution should maintain two key features of the current system:
- Live logs are be delivered to the browser in a reasonable amount time (latency should be seconds, not minutes)
- Logs are stored for long enough that if a compute node running a container fails abruptly, there is a reasonable period where an admin doing a post-mortem can access logs leading right up until the point that that the compute node went away.
Updated by Peter Amstutz almost 5 years ago
- Related to Story #16442: Scalable + reliable container logging added
Actions