Cluster configuration » History » Revision 4
Revision 3 (Tom Clegg, 06/15/2018 06:33 PM) → Revision 4/33 (Tom Clegg, 06/15/2018 07:12 PM)
h1. Cluster configuration
We are (2018) consolidating configuration from per-microservice yaml/json/ini files into a single cluster configuration document that is used by all components.
* Long term: system nodes automatically keep their configs synchronized (using something like consul).
* Short term: sysadmin uses tools like puppet and terraform to ensure /etc/arvados/config.yml is identical on all system nodes.
* Hosts without config files (e.g., hosts outside the cluster) can retrieve the config document from the API server.
h2. Discovery document
Previously, we copied selected config values from the API server config into the API discovery document so clients could see them. When clients can get the configuration document itself, this won't be needed. The discovery document should advertise APIs provided by the server, not cluster configuration.
h2. Example config file
(Format not yet frozen!)
<pre><code class="yaml">
Clusters:
xyzzy:
BlobSigningKey: ungu355able
BlobSignatureTTL: 172800
PostgreSQL: Volumes:
Host: localhost xyzzy-keep-0:
Type: s3
Region: us-east
Bucket: xyzzy-keep-0
# [rest of keepstore volume config goes here]
Providers:
Port: 5432 AWS:
# [credentials and stuff go here]
InstanceTypes:
- Name: m4.large
Username: arvados VCPUs: 2
Password: s3cr3t RAM: 8000000000
Database: arvados_production Scratch: 31000000000
Encoding: utf8 Price: 0.1
HTTPRequestTimeout: 5m - Name: m4.large-1t
# same instance type as m4.large but our scripts attach more scratch
ProviderType: m4.large
VCPUs: 2
RAM: 8000000000
Scratch: 999000000000
Price: 0.12
- Name: m4.xlarge
VCPUs: 4
RAM: 16000000000
Scratch: 78000000000
Price: 0.2
- Name: m4.8xlarge
VCPUs: 40
RAM: 160000000000
Scratch: 156000000000
Price: 2
- Name: m4.16xlarge
VCPUs: 64
RAM: 256000000000
Scratch: 310000000000
Price: 3.2
- Name: c4.large
VCPUs: 2
RAM: 3750000000
Price: 0.1
- Name: c4.8xlarge
VCPUs: 36
RAM: 60000000000
Price: 1.591
NodeProfiles:
# Key is a profile name; can be specified on service prog command line, defaults to $(hostname)
keep:
# Don’t run other services automatically -- only specified ones
Default: {Disable: true}
Keepstore: {Listen: ":25107"}
apiserver:
Default: {Disable: true}
RailsAPI: {Listen: ":9000", TLS: true}
Controller: {Listen: ":9100"}
Websocket: {Listen: ":9101"}
Health: {Listen: ":9199"}
keep:
Default: {Disable: true}
KeepProxy: {Listen: ":9102"}
KeepWeb: {Listen: ":9103"}
*:
# This section used for a node whose profile name is not listed above
Default: {Disable: false} # (this is the default behavior)
Volumes: PostgreSQL:
xyzzy-keep-0:
Type: s3
Region: us-east
Bucket: xyzzy-keep-0
# [rest of keepstore volume config goes here]
Providers: Host: localhost
AWS:
# [credentials and stuff go here] Port: 5432
Username: arvados
Password: s3cr3t
Database: arvados_production
Encoding: utf8
WebRoutes:
# “default” means route according to method/path
xyzzy.arvadosapi.com: default
# “collections” means always route to keep-web
collections.xyzzy.arvadosapi.com: collections
# leading * is a wildcard (longest match wins)
"*--collections.xyzzy.arvadosapi.com": collections
cloud.curoverse.com: workbench
workbench.xyzzy.arvadosapi.com: workbench
"*.xyzzy.arvadosapi.com": default
InstanceTypes:
- Name: m4.large
VCPUs: 2
RAM: 8000000000
Scratch: 31000000000
Price: 0.1
- Name: m4.large-1t
# same instance type as m4.large but our scripts attach more scratch
ProviderType: m4.large
VCPUs: 2
RAM: 8000000000
Scratch: 999000000000
Price: 0.12
- Name: m4.xlarge
VCPUs: 4
RAM: 16000000000
Scratch: 78000000000
Price: 0.2
- Name: m4.8xlarge
VCPUs: 40
RAM: 160000000000
Scratch: 156000000000
Price: 2
- Name: m4.16xlarge
VCPUs: 64
RAM: 256000000000
Scratch: 310000000000
Price: 3.2
- Name: c4.large
VCPUs: 2
RAM: 3750000000
Price: 0.1
- Name: c4.8xlarge
VCPUs: 36
RAM: 60000000000
Price: 1.591 HTTPRequestTimeout: 5m
</code></pre>