qcg.pilotjob.joblist module¶

class qcg.pilotjob.joblist.JobState¶

Bases: enum.Enum

The job state.

QUEUED = 1¶

SCHEDULED = 2¶

EXECUTING = 3¶

SUCCEED = 4¶

FAILED = 5¶

CANCELED = 6¶

OMITTED = 7¶

is_finished()¶: Check if job state is finished (final).

stats(stats)¶

class qcg.pilotjob.joblist.JobExecution(exec=None, args=None, env=None, script=None, stdin=None, stdout=None, stderr=None, modules=None, venv=None, wd=None, model=None, model_opts=None)¶

Bases: object

The execution element of job description.

exec¶

path to the executable

Type:	str, optional

args¶

list of arguments

Type:	list(str), optional

env¶

list of environment variables

Type:	dict(str, str), optional

stdin¶

path to the standard input file

Type:	str, optional

stdout¶

path to the standard output file

Type:	str, optional

stderr¶

path to the standard error file

Type:	str, optional

modules¶

list of modules to load before job start

Type:	list(str), optional

venv¶

path to the virtual environment to initialize before job start

Type:	str, optional

wd¶

path to the job’s working directory

Type:	str, optional

model¶

model of execution

Type:	str, optional

model_opts¶

model options

Type:	str, optional

Initialize execution element of job description.

Parameters:

exec (str, optional) – path to the executable
args (list(str), optional) – list of arguments
env (dict(str, str), optional) – list of environment variables
stdin (str, optional) – path to the standard input file
stdout (str, optional) – path to the standard output file
stderr (str, optional) – path to the standard error file
modules (list(str), optional) – list of modules to load before job start
venv (str, optional) – path to the virtual environment to initialize before job start
wd (str, optional) – path to the job’s working directory
model (str, optional) – model of execution
model_opts (dict(str, str), optional) – model options

Raises:

IllegalJobDescription – when: * nor exec or script are defined, * script and one of exec, args or env is both defined * args is not a list * env is not a dictionary

to_dict()¶

Serialize execution element to dictionary.

Returns:	dictionary with `execution` element values
Return type:	dict(str)

to_json()¶

Serialize execution element to JSON description.

Returns:	JSON description of `execution` element.

class qcg.pilotjob.joblist.ResourceSize(exact=None, min=None, max=None, scheduler=None)¶

Bases: object

The resources size element used in job description when specified the number of required cores or nodes.

Initialize resource size.

Parameters:	exact (int, optional) – exact number of resources min (int, optional) – minimum number of resources max (int, optional) – maximum number of resources scheduler (dict, optional) – the iteration resources scheduler, the `name` and `params` (optional) keys
Raises:	IllegalResourceRequirements raised when – * `exact` number and one of `min`, `max` or `scheduler` is both specified * nor `exact` or `min` or `max` is not specified * `max` and `min` is specified and `min` > `max`

exact¶

exact number of resources.

Type:	int

min¶

minimum number of resources

Type:	int

max¶

maximum number of resources

Type:	int

scheduler¶

iteration resource scheduler name

Type:	str

range¶

tuple with resources range

Type:	(int, int)

is_exact()¶

Check if resource size is defined as exact number.

Returns:	if resource size is defined as exact number False: if reosurce size is defined as range
Return type:	True

to_dict()¶

Serialize resource size to dictionary

Returns:	dictionary with resource size
Return type:	dict(str)

to_json()¶

Serialize resource size to JSON description.

Returns:	JSON description of resource size element.

class qcg.pilotjob.joblist.JobResources(numCores=None, numNodes=None, wt=None, nodeCrs=None)¶

Bases: object

The `resources` element of job description.

Initialize resources element of job description.

if numNodes > 1, then numCores relates to each of the node, so total number of

required cores will be a product of numNodes and numCores
nodeCrs relates to each node available consumable resources

Parameters:	- number of cores, either as exact number or as a range (numCores) – - number of nodes, either as exact number of as a range (numNodes) – - wall time (wt) – nodeCrs (dict(string,int)) –
Raises:	`IlleglResourceRequirements` – raised when: * `numCores` and `numNodes` not defined * `numCores` or `numNodes` not instance of either `int`, `dict` or ResourceSize * wrong consumable resources definition

has_nodes¶

true if resources element of job description contains number of nodes definition

Type:	bool

has_cores¶

true if resources element of job description contains number of cores definition

Type:	bool

has_crs¶

true if resources element of job description contains consumable resources definition

Type:	bool

cores¶

return numCores definition of resources element.

Type:	ResourceSize

nodes¶

return numNodes definition of resources element.

Type:	ResourceSize

crs¶

return nodeCrs definition of resources element.

Type:	ResourceSize

get_min_num_cores()¶

Return minimum number of cores the job can be run.

Returns:	minimum number of required cores for the job.
Return type:	int

to_dict()¶

Serialize resource element of job description to dictionary

Returns:	dictionary with `resources` element of job description
Return type:	dict(str)

to_json()¶

Serialize resource element of job description to JSON description.

Returns:	JSON description of `resource` element of job description.

class qcg.pilotjob.joblist.JobDependencies(after=None)¶

Bases: object

Runtime dependencies of job.

Initialize runtime dependencies of a job.

Parameters:	- list of jobs that must finish before job can be started (after) –
Raises:	`IllegalJobDescription` – when list of jobs has a wrong format.

has_dependencies¶

true if job contains runtime dependencies

Type:	bool

to_dict()¶

Serialize job’s runtime dependencies

Returns:	dictionary with job’s runtime dependencies
Return type:	dict(str)

to_json()¶

Serialize job’s runtime dependencies to JSON description.

Returns:	JSON description of job’s runtime dependencies

class qcg.pilotjob.joblist.JobIteration(start=None, stop=None, values=None)¶

Bases: object

The iteration element of job description.

Initialize iteration element of job description.

The iteration can be defined as a range with start` and `stop parameters, or as an values set. If start is not defined but stop is, the value 0 is assumed as a start.

Parameters:	start (int) – starting index of an iteration stop (int) – stop index of an iteration - the last value of job’s iteration will be `stop` - 1 values (list(str)) – the enumerated list of iteration values
Raises:	`IllegalJobDescription` – raised when: * `stop` and `values` is not defined * `stop` and `values` is both defined * `start` is greater or equal `stop`

in_range(index)¶

Check if given index is in range of job’s iterations.

Parameters:	index (int) – index to check
Returns:	true if index is in range
Return type:	bool

iterations_gen()¶

Iterations generator.

Returns:	the iteration indexes
Return type:	int

iterations()¶

Return number of iterations of a job.

Returns:	number of iterations
Return type:	int

iteration_value(index)¶

Return value related with the iteration index.

Returns:	if iteration has been defined with `values` argument the value on position according to the iteration index will be returned, oterwise the string representation of index will be returned
Return type:	str

to_dict()¶

Serialize iteration element of job description

Returns:	dictionary with `iteration` element of job description
Return type:	dict(str)

to_json()¶

Serialize iteration element of job description to JSON description.

Returns:	JSON description of `iteration` element of job description

class qcg.pilotjob.joblist.SubJobState¶

Bases: object

Represent state of execution of single job’s iteration.

Initialize state of execution of single job’s iteration.

The initial state is set to QUEUED and all other attributes are initialized as empty elements.

state()¶

Return current status of job’s iteration.

Returns:	current status of job’s iteration
Return type:	JobState

set_state(state, err_msg=None)¶

Set current state of job’s iteration.

Parameters:	state (JobState) – the new state of job’s iteration err_msg (str, optional) – the error message to record

append_runtime(data)¶

Record job’s iteration runtime statistics.

Parameters:	data (dict) – the data to append to job’s iteration runtime statistics

history()¶

Return job’s iteration state change history.

Returns:	job’s iteration state change history.
Return type:	list(JobState, DateTime)

messages()¶

Return job’s iteration recorded error messages.

Returns:	recorded job’s iteration error messages.
Return type:	list(str)

runtime()¶

Return job’s iteration runtime statistics.

Returns:	job’s iteration runtime statistics
Return type:	dict

append_message(msg)¶

Record job’s iteration error message.

Parameters:	msg (str) – error message to record

class qcg.pilotjob.joblist.Job(name, execution, resources, iteration=None, dependencies=None, attributes=None)¶

Bases: object

Job description and state.

_name¶

job name

Type:	str

_execution¶

execution description

Type:	JobExecution

_resources¶

resources description

Type:	JobResources

_iteration¶

iteration description

Type:	JobIteration

dependencies¶

runtime dependencies description

Type:	JobDependencies

attributes¶

additional attributes

Type:	dict

_subjobs¶

list of job’s iteration states - only if iteration element defined, elements at positions ‘job iteration’ - ‘iteration start’

Type:	list(SubJobState)

_subjobs_not_finished¶

number of not finished already iterations - only if iteration element defined

Type:	int

_subjobs_failed¶

number of already failed iterations - only if iteration element defined

Type:	int

_history¶

state change history

Type:	list(JobState, DateTime)

_state¶

current state

Type:	JobState

_messages¶: recorded error messages

_runtime¶

runtime information

Type:	dict

_queue_pos¶: current job’s position in scheduling queue

Initialize job.

Parameters:

name (str) – job name
execution (JobExecution or dict) – execution element of job’s description
resources (JobResources or dict) – resources element of job’s description
iteration (JobIteration or dict, optional) – iteration element of job’s description
dependencies (JobDependencies or dict, optional) – dependencies element of job’s description
attributes (dict, optional) – additional job’s attributes used by partition manager

Raises:

IllegalJobDescription – raised in case of wrong elements of job’s description

static validate_jobname(jobname)¶

Check if given name is valid for job’s name.

Parameters:	jobname (str) – name to validate
Returns:	true if name is valid
Return type:	bool

name¶

job’s name

Type:	str

get_name(iteration=None)¶

Return job’s or job’s iteration name.

Parameters:	iteration (int, optional) – if defined the iteration’s name is returned
Returns:	job’s or job’s iteration’s name
Return type:	str

execution¶

the execution element of job description

Type:	JobExecution

resources¶

the resources element of job description

Type:	JobExecution

get_not_finished_iterations()¶

Return number of currently not finished iterations.

This method is valid only for iteration jobs.

Returns:	number of not finished iterations
Return type:	int

get_failed_iterations()¶

Return number of already failed iterations.

This method is valid only for iteration jobs.

Returns:	number of failed iterations
Return type:	int

history(iteration=None)¶

Return job’s or job’s iteration state change history.

Parameters:	iteration (int, optional) – if defined the iteration’s state change history is returned
Returns:	job’s or job’s iteration state change history.
Return type:	list(JobState, DateTime)

messages(iteration=None)¶

Return job’s or job’s iteration recorded error messages.

Parameters:	iteration (int, optional) – if defined the iteration’s recorded error messages is returned
Returns:	recorded job’s or job’s iteration error messages.
Return type:	list(str)

runtime(iteration=None)¶

Return job’s or job’s iteration runtime statistics.

Parameters:	iteration (int, optional) – if defined the iteration’s runtime statistics is returned
Returns:	job’s or job’s iteration runtime statistics
Return type:	dict

has_iterations¶

true if job has iterations

Type:	bool

iteration¶

iteration element of job description

Type:	JobIteration

state(iteration=None)¶

Return job’s or job’s iteration current state.

Parameters:	iteration (int, optional) – if defined the iteration’s state is returned
Returns:	job’s or job’s iteration current state
Return type:	JobState

str_state(iteration=None)¶

Return job’s or job’s iteration current state as string.

Parameters:	iteration (int, optional) – if defined the iteration’s state is returned
Returns:	job’s or job’s iteration current state as string
Return type:	JobState

iteration_states¶

list of iteration states

Type:	list(SubJobState)

set_state(state, iteration=None, err_msg=None)¶

Set current job’s or job’s iteration state.

Parameters:

state (JobState) – new job’s or job’s iteration state
iteration (int, optional) – job’s iteration index if the iteration state should be set
err_msg (str, optional) – the optional error message to record

Returns:

the job’s finish state if job’s iteration status change triggered job status change (for example: the last iteration job finished, so the whole job also finished), or None if job’s as a whole still not finished

Return type:

JobState

has_dependencies¶

true if job has runtime dependencies

Type:	bool

append_message(msg)¶

Record job’s error message.

Parameters:	msg (str) – error message to record

queue_pos()¶

Return current position of a job in scheduling queue

Returns:	current position of a job in scheduling queue
Return type:	int

set_queue_pos(pos)¶

Set current position of a job in scheduling queue.

Parameters:	pos (int) – current position of a job in scheduling queue

clear_queue_pos()¶: Reset current position of a job in scheduling queue.

append_runtime(data, iteration)¶

Record job’s or job’s iteration runtime statistics.

Parameters:	data (dict) – the data to append to job’s or job’s iteration runtime statistics iteration (int, optional) – if defined the iteration’s runtime statistics will be updated data – the data to append to job’s or job’s iteration runtime statistics

to_dict()¶

Serialize job’s description to dictionary.

Returns:	dictionary with job description
Return type:	dict(str)

to_json()¶

Serialize job description to JSON format.

Returns:	JSON of job’s description

class qcg.pilotjob.joblist.JobList¶

Bases: object

The list of all submited jobs.

_jmap¶

dictionary with all submited jobs with name as key

Type:	dict(str,Job)

Initialize the list.

static parse_jobname(jobname)¶

Split given name into job name and iteration.

Parameters:	jobname (str) – the name to parse
Returns:	tuple with job name and iteration, if given job name didn’t contain iteration index, the second value will be None
Return type:	name, iteration

add(job)¶

Add a new job.

Parameters:	job (Job) – job to add to the list
Raises:	`JobAlreadyExist` – when job with given name already exists in list

exist(jobname)¶

Check if job with given name is in list.

Parameters:	jobname (str) – job name to check
Returns:	true if job with given name is already in list
Return type:	bool

get(jobname)¶

Return job with given name.

Parameters:	jobname (str) – job name
Returns:	job from the list or None if not such job has been added.
Return type:	Job

jobs()¶

Return all job names in the list.

Returns:	set-like object with job names

remove(jobname)¶

Remove job with given name from list.

Parameters:	jobname (str) – job’s name to remove from list