Arvados

h1. Glossary of Arvados Terms 

*collection* - snapshot of a set of data files (file names and contents). Analogous to a directory tree, with the additional property that it reliably identifies the data itself, not just symbolic names.

*content address* - a cryptographic digest of a data blob that can be used to determine where the data is stored and to verify the file.

*data manager* - a component that assists Keep servers in enforcing site policies and monitors the state of the storage facility as a whole; we expect that the Data Manager will also broker backup and archival services and interfaces to other external data stores.

*job dispatcher* - invokes jobs from the queue, as compute nodes become available.

*job script* - a program written and organized to be implemented using MapReduce for distributed computation.

*job task* - a portion of a Job that can be computed asynchronously and independently of other Job Tasks; could also be called a “MapReduce task” or “MapReduce step”.

*job* - an execution of a MapReduce program (Job Script). Has specific inputs, outputs, start time, finish time, etc.

*Keep* - a content addressable distributed file system.

*manifest* - a plain-text encoding of a collection. Lists filenames, sizes, and data block locators (hashes). Reference: http://en.wikipedia.org/wiki/Manifest_file.

*MapReduce Engine* - supervises a job from start to finish: sets up execution environment, queues and executes tasks, monitors task/node failures, reports statistics.

*pipeline component* - a portion of a pipeline template (or instance) that describes criteria for selecting or submitting a job; can also refer to a specific Job (e.g., in a completed pipeline instance).

*pipeline instance* - the act or record of applying a pipeline template to a specific set of inputs; generally, a pipeline instance refers to the UUIDs of jobs that have been run to satisfy the pipeline components.

*pipeline manager* - looks up (and submits, as needed) jobs to satisfy pipeline components, monitors the jobs as they run, and handles dependencies (e.g., wait for job A to complete, and use its output as job B’s input).

*pipeline template* - a pattern that describes the relationships among the component Jobs: for example, the template specifies that job A's output is job B's input; a pipeline template is analogous to a Makefile.

*provenance* - the origin of data and the details of how a data set was computed (including processes and source data).

*Workbench* - the browser-based visual tools for users to interface with Arvados capabilities; Workbench mirrors many of the same functions accessible through the command line interface (CLI).