Jug Master Database Schema

The tables and columns that make up the master database are listed below.

batch

This table contains one entry per batch. It defines information common to all jobs in the batch, such as software packages to use, environment variables, and the output path.

batch_id	e.g. 3
source	e.g. "RefDB"
source_batch_id	e.g. RefDB batch ID
name	e.g. "eg02_BitJets-1974"
type	"joblist" or "seed"
rank	Relative weight for computing resource share.
job_runtime	Expected runtime (seconds)
environment	variable=value lines (newline escaped = '\n')
software	List of URLs (delimited by newline). See software.
workware	Just like software packages, but installed in the job's working directory.
input_files	Files installed in the working directory. No unpacking of archives, by default.
execution_class	Execution workers to use.
storage_class	Output storage workers to use.
output_path	Where to put output files.
err_storage_class	Handler(s) to send failed jobs to.
err_output_path	Where to put failed output data.
started	When batch first started.
finished	When batch was completed.
paused	When batch was suspended.
next_run_id	Used to generate unique run ids
next_job_id	Used to generate unique job ids
work_done	Amount of work done (events, iterations, etc.) if reported by jobs
output_bytes	Total size of output files stored.
notes	Free-form text field.
failure_throttle	Automatic throttle when rapid failure state is observed

job

This table provides a list of jobs belonging to a batch. For batch.type of "seed", entries are automatically created using information from the seed table.

batch_id	e.g. "1974"
job_id
seed	the seed number, if applicable
workware	like batch workware but not cached by default
input_files	List of URLs. Transfers will not be cached.
environment	adds onto batch environment
output_path	relative to batch output path
runs_started	count of run attempts
last_started	time of last attempt
ETA	estimated time run will finish
finished	Date of completion.
committing	Date when execution and stage-out finished (worker_finished).
on_hold	Date when job was put on hold.
failed	Date when job failed.
failure_code	Numerical error code.
failure_name	Error name.
failure_desc	Full description of error.
work_done	Amount of work done (events, iterations, etc.) if reported by jobs
output_bytes	Total size of output files stored.

seed

This table is used to define the seed range for batchs of type "seed". Each job is allocated a single seed number from the specified range.

batch_id	e.g. "1974"
seed_low	Starting value for random seed.
seed_high	Ending value. -1 if unlimited.
seed_next	Next seed to use.
finished	Date of completion.

run

Each entry in this table corresponds to a job that is running or that did run on a worker somewhere. Job failures may result in multiple entries for the same underlying task, but each one will have its own unique id.

batch_id
run_id
job_id
worker_type	'E' for execution, 'S' for storage, 'Q' for queue
handling_run_id	run_id being stored or queued
handling_file_id	for storage jobs; file_id being stored
cookie	magic number to authenticate worker
environment	Final environment (with any substitutions made).
all_input_files	Software, workware, input_files, distinguished by (type=X).
started	Date/time when started.
finished	Date/time successfully completed (both worker and storage handlers signed off).
worker_finished	Date/time worker completed.
ETA	Expected time of arrival.
last_heard_from
lease_expires
IP	IP address of worker (may shift)
worker_url	URL of worker HTTP socket, if any.
storage_id	Storage handler running this job.
output_path	Final output_path (with any substitions made).
output_files	Output files received.
err_output_path	Where failed data was stored.
err_output_files
failed	Date/time of failure.
failure_code	Numerical error code.
failure_name	Name of error code.
failure_desc	Extra information describing failure.
get_job_time	time spent in GetJob RPC call
stage_in_time	time spent staging in files
run_time	time spent running the job
stage_out_time	time spent in CommitJob RPC call
stage_out_idle	time spent waiting to poll status
work_done	Amount of work done (events, iterations, etc.) if reported by jobs
state	State of execution in JugWorker
waiting	Non-zero if waiting to enter new state

run_output

This table contains an entry for each file produced by a run.

batch_id
run_id
file_id	Id of file (unique within run).
cookie	Storage authorization cookie.
size	Size of file.
md5	MD5 checksum of file.
source_name	Name of file in worker output area.
source_url	How to get file from worker.
storage_id	Storage handler.
storage_name	Name/path of file in storage.
storage_hfn	host file name: host.name:/path/to/file
storage_url	URL of file in storage
created	Date and time file was created.
fetched	Date and time file was transferred from worker that generated it.
committed	Date and time file was committed to storage
ETA	estimated time to fetch

worker_class

Each entry in this table defines a family of workers which are all authenticated and configured the same way.

worker_class	class name
worker_type	"E", "S", or "Q"
parent_class	worker_class of parent
queue_class	Mame matched by job_selector of a queue worker.
job_selector	execution, storage, or queue class
cookie	Authentication cookie.
software	Extra software.
environment	Extra environment variables.
runtime_options	List of worker runtime options.
queue_options	Options passed to queue job that creates this worker.
base_output_path	Base path/URL where files are stored if this is a storage worker.
created	Date of creation of this record.

worker

Each entry in this table represents an active worker.

worker_id	unique id
worker_class	execution or storage class
worker_type	"E" or "S"
cookie	Authentication cookie.
created	Date of creation of this record.
last_heard_from	Time of last contact.
IP	Address of host.
worker_url	Contact URL.

file_server

Each entry in this table maps a URL to a base HFN (host file name). URLs of the form hfn://host:path are automatically translated into a URL that accesses the file server.

name	unique name
server_url	base URL to be substitued for base HFN
base_hfn	hfn://host.name:/path/to/file
created	time this record was created
lease_expires	presume dead unless we hear sooner
cookie	Authorization cookie.

security_log

This table holds a record of authorization failures and the like.

time	Date and time of event.
IP	IP address of client.
batch_id	The batch_id that was presented.
run_id	The run_id that was presented.
run_cookie	Authorization cookie that was presented.
worker_id	ID of worker.
worker_cookie	Authentication cookie that was presented.
worker_class	Worker class name.
worker_class_cookie	Authentication cookie that was presented.
activity	Description of rejected request.

master_rpc

This table contains one entry for each master SOAP RPC handler.

url	URL of the handler.
created	Date when created.
last_heard_from	Date when last heard from.
expires	Date when should presume dead.

);