Arvados

As of 64efd10 here is the output of keepstore -help:

keepstore provides a content-addressed data store backed by a local filesystem or networked storage.

Usage: keepstore -config path/to/keepstore.yml
       keepstore [OPTIONS] -dump-config

NOTE: All options (other than -config) are deprecated in favor of YAML
      configuration. Use -dump-config to translate existing
      configurations to YAML format.

Options:
  -azure-max-get-bytes int
        Maximum bytes to request in a single GET request. If smaller than 67108864, use multiple concurrent range requests to retrieve a block. (default 67108864)
  -azure-storage-account-key-file File
        File containing the account key used for subsequent --azure-storage-container-volume arguments.
  -azure-storage-account-name string
        Azure storage account name used for subsequent --azure-storage-container-volume arguments.

[...all previously existing options are here...]

  -dump-config
        write current configuration to stdout and exit (useful for migrating from command line flags to config file)

[...]

  -trash-check-interval value
        see TrashCheckInterval configuration (default 24h0m0s)
  -trash-lifetime value
        see TrashLifetime configuration (default 336h0m0s)
  -volume value
        see Volumes configuration
  -volumes value
        see Volumes configuration

Example config file:

BlobSignatureTTL: 336h0m0s
BlobSigningKeyFile: "" 
EnableDelete: false
Listen: :25107
MaxBuffers: 128
MaxRequests: 0
PIDFile: "" 
RequireSignatures: true
SystemAuthTokenFile: "" 
TrashCheckInterval: 24h0m0s
TrashLifetime: 336h0m0s
Volumes:
- AzureReplication: 3
  ContainerName: example-container-name
  ReadOnly: false
  StorageAccountKeyFile: /etc/azure_storage_account_key.txt
  StorageAccountName: example-account-name
  Type: Azure
- AccessKeyFile: /etc/aws_s3_access_key.txt
  Bucket: example-bucket-name
  Endpoint: "" 
  IndexPageSize: 1000
  LocationConstraint: false
  RaceWindow: 24h0m0s
  ReadOnly: false
  Region: us-east-1
  S3Replication: 2
  SecretKeyFile: /etc/aws_s3_secret_key.txt
  Type: S3
  UnsafeDelete: false
- AccessKeyFile: /etc/gce_s3_access_key.txt
  Bucket: example-bucket-name
  Endpoint: https://storage.googleapis.com
  IndexPageSize: 1000
  LocationConstraint: false
  RaceWindow: 24h0m0s
  ReadOnly: false
  Region: "" 
  S3Replication: 2
  SecretKeyFile: /etc/gce_s3_secret_key.txt
  Type: S3
  UnsafeDelete: false
- ReadOnly: false
  Root: /mnt/local-disk
  Serialize: true
  Type: Directory
- ReadOnly: false
  Root: /mnt/network-disk
  Serialize: false
  Type: Directory

Listen:

    Local port to listen on. Can be "address", "address:port", or
    ":port", where "address" is a host IP address or name and "port" 
    is a port number or name.

PIDFile:

   Path to write PID file during startup. This file is kept open and
   locked with LOCK_EX until keepstore exits, so "fuser -k pidfile" is
   one way to shut down. Exit immediately if there is an error
   opening, locking, or writing the PID file.

MaxBuffers:

    Maximum RAM to use for data buffers, given in multiples of block
    size (64 MiB). When this limit is reached, HTTP requests requiring
    buffers (like GET and PUT) will wait for buffer space to be
    released.

MaxRequests:

    Maximum concurrent requests. When this limit is reached, new
    requests will receive 503 responses. Note: this limit does not
    include idle connections from clients using HTTP keepalive, so it
    does not strictly limit the number of concurrent connections. If
    omitted or zero, the default is 2 * MaxBuffers.

BlobSigningKeyFile:

    Local file containing the secret blob signing key (used to
    generate and verify blob signatures).  This key should be
    identical to the API server's blob_signing_key configuration
    entry.

RequireSignatures:

    Honor read requests only if a valid signature is provided.  This
    should be true, except for development use and when migrating from
    a very old version.

BlobSignatureTTL:

    Duration for which new permission signatures (returned in PUT
    responses) will be valid.  This should be equal to the API
    server's blob_signature_ttl configuration entry.

SystemAuthTokenFile:

    Local file containing the Arvados API token used by keep-balance
    or data manager.  Delete, trash, and index requests are honored
    only for this token.

EnableDelete:

    Enable trash and delete features. If false, trash lists will be
    accepted but blocks will not be trashed or deleted.

TrashLifetime:

    Time duration after a block is trashed during which it can be
    recovered using an /untrash request.

TrashCheckInterval:

    How often to check for (and delete) trashed blocks whose
    TrashLifetime has expired.

Volumes:

    List of storage volumes. If omitted or empty, the default is to
    use all directories named "keep" that exist in the top level
    directory of a mount point at startup time.

here's -dump-config from keep0.4xphq

BlobSignatureTTL: 336h0m0s
BlobSigningKeyFile: /etc/keepstore/permission.key
EnableDelete: false
Listen: :25107
MaxBuffers: 60
MaxRequests: 256
PIDFile: "" 
RequireSignatures: true
SystemAuthTokenFile: /etc/keepstore/datamanager.key
TrashCheckInterval: 24h0m0s
TrashLifetime: 168h0m0s
Volumes:
- AccessKeyFile: /etc/keepstore/amazon_s3_access.key
  Bucket: 4xphq-keep
  Endpoint: "" 
  IndexPageSize: 1000
  LocationConstraint: false
  RaceWindow: 24h0m0s
  ReadOnly: false
  Region: us-east-1
  S3Replication: 2
  SecretKeyFile: /etc/keepstore/amazon_s3_secret.key
  Type: S3
  UnsafeDelete: false

• document fields for each volume type (where should this go?)
• show the config field descriptions as YAML comments in example config (and -dump-config)

• Wondering if “Directory” is clear enough to mean “unix file system” for the unix volume type.

• Listen: If just “address” is configured, what would the port be? 25107? Can you please list the default here, if there is any?

• Wondering if we could use the word “Template” instead of “Example” in VolumeWithExamples etc?

• Since all other config option other than “config” are deprecated, I think moving the following to the top of “main” in keepstore.go would make the code easier to understand and maintain in the long run

    defaultConfigPath := "/etc/arvados/keepstore/keepstore.yml" 
    var configPath string
    flag.StringVar(
        &configPath,
        "config",
        defaultConfigPath,
        "YAML or JSON configuration file `path`")

• Can you move these to close to their definitions?
  

      neverDelete := !theConfig.EnableDelete
      signatureTTLSeconds := int(theConfig.BlobSignatureTTL.Duration() / time.Second)
  

• “ // TODO: Load config ” -- still need "TODO" here?

• usage.go
  • I think "Volumes" needs to include the supported volume types and an example with explanation under each type. Currently, you would need to read through the "deprecated" options to get the whole picture

• When looking at "keepstore -h" , since all but options other than "config" are deprecated anyways:
  • moving the "Options" all the way to the bottom after all the other explanation
  • It might help if we move the Example config to the top, right after the "Note"
  • adding comments to the yaml would help (and probably eliminate the need to explain them in Volumes)

• Regarding the keepstore.service file: http://www.yaml.org/faq.html seems to suggest using .yaml when possible, and am wondering if we should use that instead if .yml

This became a blocker for us. Can we raise the priority of this issue? We'll need a installable package with configuration and systemd file in place.

Radhika Chippada wrote:

• Wondering if “Directory” is clear enough to mean “unix file system” for the unix volume type.

I think so. Saying something like "unix file system" might imply we take over an entire filesystem.

• Listen: If just “address” is configured, what would the port be? 25107? Can you please list the default here, if there is any?

Oops, that's not actually possible. Removed that.

• Wondering if we could use the word “Template” instead of “Example” in VolumeWithExamples etc?

I think "example" is more accurate...?

• Since all other config option other than “config” are deprecated, I think moving the following to the top of “main” in keepstore.go would make the code easier to understand and maintain in the long run

[...]

• Can you move these to close to their definitions?
  [...]

Yeah. Moved all that deprecated stuff to deprecated.go

• “ // TODO: Load config ” -- still need "TODO" here?

Oops, removed.

• usage.go
  • I think "Volumes" needs to include the supported volume types and an example with explanation under each type. Currently, you would need to read through the "deprecated" options to get the whole picture

Added the list of types, because that's pretty easy. Explaining all the volume (and other) configs in a convenient way will require a bit more work and I don't think it needs to be a blocker.

• When looking at "keepstore -h" , since all but options other than "config" are deprecated anyways:
  • moving the "Options" all the way to the bottom after all the other explanation
  • It might help if we move the Example config to the top, right after the "Note"
  • adding comments to the yaml would help (and probably eliminate the need to explain them in Volumes)

Yes, I think a well-commented example config is the way to go. Then the help message can reduce to just "-config", the deprecated options, and maybe another flag that prints out the commented example config?

• Regarding the keepstore.service file: http://www.yaml.org/faq.html seems to suggest using .yaml when possible, and am wondering if we should use that instead if .yml

Hmmm... valid point, but unfortunately at this point we've already committed to *.yml with other services, so I think we should go with internal consistency until we come up with a motivation/migration across the board.

• (mostly) LGTM. Please merge if Nico is waiting.

Before marking the ticket as resolved though, I think it would be great if you could address:

Yes, I think a well-commented example config is the way to go. Then the help message can reduce to just "-config", the deprecated options, and maybe another flag that prints out the commented example config?

However, I think no need to add another flag to print the example config and no harm in printing it like now (but whatever you like).

• Also, please consider separating the "Options" display into two sections. "Options (scroll down to see deprecated)" where just the "-config" is listed and "Deprecated Options" or "All Options including deprecated" at the bottom of the command display. Or a flag to list the deprecated.

Thanks.

is there a simple way to expand this feature to have a -verify-config too ? before restarting services willl be good to automatically check that the config doesn't have syntax errors.