qcg.pilotjob.joblist module

class qcg.pilotjob.joblist.JobState(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

The job state.

QUEUED = 1

SCHEDULED = 2

EXECUTING = 3

SUCCEED = 4

FAILED = 5

CANCELED = 6

OMITTED = 7

is_finished()

Check if job state is finished (final).

stats(stats)

class qcg.pilotjob.joblist.JobExecution(exec=None, args=None, env=None, script=None, stdin=None, stdout=None, stderr=None, modules=None, venv=None, wd=None, model=None, model_opts=None)

Bases: object

The execution element of job description.

exec

path to the executable

Type

str, optional

args

list of arguments

Type

list(str), optional

env

list of environment variables

Type

dict(str, str), optional

stdin

path to the standard input file

Type

str, optional

stdout

path to the standard output file

Type

str, optional

stderr

path to the standard error file

Type

str, optional

modules

list of modules to load before job start

Type

list(str), optional

venv

path to the virtual environment to initialize before job start

Type

str, optional

wd

path to the job’s working directory

Type

str, optional

model

model of execution

Type

str, optional

model_opts

model options

Type

str, optional

Initialize execution element of job description.

Parameters

• exec (str, optional) – path to the executable

• args (list(str), optional) – list of arguments

• env (dict(str, str), optional) – list of environment variables

• stdin (str, optional) – path to the standard input file

• stdout (str, optional) – path to the standard output file

• stderr (str, optional) – path to the standard error file

• modules (list(str), optional) – list of modules to load before job start

• venv (str, optional) – path to the virtual environment to initialize before job start

• wd (str, optional) – path to the job’s working directory

• model (str, optional) – model of execution

• model_opts (dict(str, str), optional) – model options

Raises

IllegalJobDescription – when: * nor exec or script are defined, * script and one of exec, args or env is both defined * args is not a list * env is not a dictionary

to_dict()

Serialize execution element to dictionary.

Returns

dictionary with execution element values

Return type

dict(str)

to_json()

Serialize execution element to JSON description.

Returns

JSON description of execution element.

class qcg.pilotjob.joblist.ResourceSize(exact=None, min=None, max=None, scheduler=None)

Bases: object

The resources size element used in job description when specified the number of required cores or nodes.

Initialize resource size.

Parameters

• exact (int, optional) – exact number of resources

• min (int, optional) – minimum number of resources

• max (int, optional) – maximum number of resources

• scheduler (dict, optional) – the iteration resources scheduler, the name and params (optional) keys

Raises

IllegalResourceRequirements raised when –

• exact number and one of min, max or scheduler is both specified * nor exact or min or max is not specified * max and min is specified and min > max

property exact

exact number of resources.

Type

int

property min

minimum number of resources

Type

int

property max

maximum number of resources

Type

int

property scheduler

iteration resource scheduler name

Type

str

property range

tuple with resources range

Type

(int, int)

is_exact()

Check if resource size is defined as exact number.

Returns

if resource size is defined as exact number False: if reosurce size is defined as range

Return type

True

to_dict()

Serialize resource size to dictionary

Returns

dictionary with resource size

Return type

dict(str)

to_json()

Serialize resource size to JSON description.

Returns

JSON description of resource size element.

class qcg.pilotjob.joblist.JobResources(numCores=None, numNodes=None, wt=None, nodeCrs=None)

Bases: object

The `resources` element of job description.

Initialize resources element of job description.

• if numNodes > 1, then numCores relates to each of the node, so total number of

  required cores will be a product of numNodes and numCores

• nodeCrs relates to each node available consumable resources

Parameters

• cores (numCores - number of) –

• range (either as exact number of as a) –

• nodes (numNodes - number of) –

• range –

• time (wt - wall) –

• nodeCrs (dict(string,int)) –

Raises

IlleglResourceRequirements – raised when: * numCores and numNodes not defined * numCores or numNodes not instance of either int, dict or ResourceSize * wrong consumable resources definition

property has_nodes

true if resources element of job description contains number of nodes definition

Type

bool

property has_cores

true if resources element of job description contains number of cores definition

Type

bool

property has_crs

true if resources element of job description contains consumable resources definition

Type

bool

property cores

return numCores definition of resources element.

Type

ResourceSize

property nodes

return numNodes definition of resources element.

Type

ResourceSize

property crs

return nodeCrs definition of resources element.

Type

ResourceSize

get_min_num_cores()

Return minimum number of cores the job can be run.

Returns

minimum number of required cores for the job.

Return type

int

to_dict()

Serialize resource element of job description to dictionary

Returns

dictionary with resources element of job description

Return type

dict(str)

to_json()

Serialize resource element of job description to JSON description.

Returns

JSON description of resource element of job description.

class qcg.pilotjob.joblist.JobDependencies(after=None)

Bases: object

Runtime dependencies of job.

Initialize runtime dependencies of a job.

Parameters

started (after - list of jobs that must finish before job can be) –

Raises

IllegalJobDescription – when list of jobs has a wrong format.

property has_dependencies

true if job contains runtime dependencies

Type

bool

to_dict()

Serialize job’s runtime dependencies

Returns

dictionary with job’s runtime dependencies

Return type

dict(str)

to_json()

Serialize job’s runtime dependencies to JSON description.

Returns

JSON description of job’s runtime dependencies

class qcg.pilotjob.joblist.JobIteration(start=None, stop=None, values=None)

Bases: object

The iteration element of job description.

Initialize iteration element of job description.

The iteration can be defined as a range with start` and `stop parameters, or as an values set. If start is not defined but stop is, the value 0 is assumed as a start.

Parameters

• start (int) – starting index of an iteration

• stop (int) – stop index of an iteration - the last value of job’s iteration will be stop - 1

• values (list(str)) – the enumerated list of iteration values

Raises

IllegalJobDescription – raised when: * stop and values is not defined * stop and values is both defined * start is greater or equal stop

in_range(index)

Check if given index is in range of job’s iterations.

Parameters

index (int) – index to check

Returns

true if index is in range

Return type

bool

iterations_gen()

Iterations generator.

Returns

the iteration indexes

Return type

int

iterations()

Return number of iterations of a job.

Returns

number of iterations

Return type

int

iteration_value(index)

Return value related with the iteration index.

Returns

if iteration has been defined with values argument the value on position according to the iteration

index will be returned, oterwise the string representation of index will be returned

Return type

str

to_dict()

Serialize iteration element of job description

Returns

dictionary with iteration element of job description

Return type

dict(str)

to_json()

Serialize iteration element of job description to JSON description.

Returns

JSON description of iteration element of job description

class qcg.pilotjob.joblist.SubJobState

Bases: object

Represent state of execution of single job’s iteration.

Initialize state of execution of single job’s iteration.

The initial state is set to QUEUED and all other attributes are initialized as empty elements.

state()

Return current status of job’s iteration.

Returns

current status of job’s iteration

Return type

JobState

set_state(state, err_msg=None)

Set current state of job’s iteration.

Parameters

• state (JobState) – the new state of job’s iteration

• err_msg (str, optional) – the error message to record

append_runtime(data)

Record job’s iteration runtime statistics.

Parameters

data (dict) – the data to append to job’s iteration runtime statistics

history()

Return job’s iteration state change history.

Returns

job’s iteration state change history.

Return type

list(JobState, DateTime)

messages()

Return job’s iteration recorded error messages.

Returns

recorded job’s iteration error messages.

Return type

list(str)

runtime()

Return job’s iteration runtime statistics.

Returns

job’s iteration runtime statistics

Return type

dict

append_message(msg)

Record job’s iteration error message.

Parameters

msg (str) – error message to record

class qcg.pilotjob.joblist.Job(name, execution, resources, iteration=None, dependencies=None, attributes=None)

Bases: object

Job description and state.

_name

job name

Type

str

_execution

execution description

Type

JobExecution

_resources

resources description

Type

JobResources

_iteration

iteration description

Type

JobIteration

dependencies

runtime dependencies description

Type

JobDependencies

attributes

additional attributes

Type

dict

_subjobs

list of job’s iteration states - only if iteration element defined, elements at positions ‘job iteration’ - ‘iteration start’

Type

list(SubJobState)

_subjobs_not_finished

number of not finished already iterations - only if iteration element defined

Type

int

_subjobs_failed

number of already failed iterations - only if iteration element defined

Type

int

_history

state change history

Type

list(JobState, DateTime)

_state

current state

Type

JobState

_messages

recorded error messages

_runtime

runtime information

Type

dict

_queue_pos

current job’s position in scheduling queue

Initialize job.

Parameters

• name (str) – job name

• execution (JobExecution or dict) – execution element of job’s description

• resources (JobResources or dict) – resources element of job’s description

• iteration (JobIteration or dict, optional) – iteration element of job’s description

• dependencies (JobDependencies or dict, optional) – dependencies element of job’s description

• attributes (dict, optional) – additional job’s attributes used by partition manager

Raises

IllegalJobDescription – raised in case of wrong elements of job’s description

static validate_jobname(jobname)

Check if given name is valid for job’s name.

Parameters

jobname (str) – name to validate

Returns

true if name is valid

Return type

bool

property name

job’s name

Type

str

get_name(iteration=None)

Return job’s or job’s iteration name.

Parameters

iteration (int, optional) – if defined the iteration’s name is returned

Returns

job’s or job’s iteration’s name

Return type

str

property execution

the execution element of job description

Type

JobExecution

property resources

the resources element of job description

Type

JobExecution

get_not_finished_iterations()

Return number of currently not finished iterations.

This method is valid only for iteration jobs.

Returns

number of not finished iterations

Return type

int

get_failed_iterations()

Return number of already failed iterations.

This method is valid only for iteration jobs.

Returns

number of failed iterations

Return type

int

history(iteration=None)

Return job’s or job’s iteration state change history.

Parameters

iteration (int, optional) – if defined the iteration’s state change history is returned

Returns

job’s or job’s iteration state change history.

Return type

list(JobState, DateTime)

messages(iteration=None)

Return job’s or job’s iteration recorded error messages.

Parameters

iteration (int, optional) – if defined the iteration’s recorded error messages is returned

Returns

recorded job’s or job’s iteration error messages.

Return type

list(str)

runtime(iteration=None)

Return job’s or job’s iteration runtime statistics.

Parameters

iteration (int, optional) – if defined the iteration’s runtime statistics is returned

Returns

job’s or job’s iteration runtime statistics

Return type

dict

property has_iterations

true if job has iterations

Type

bool

property iteration

iteration element of job description

Type

JobIteration

state(iteration=None)

Return job’s or job’s iteration current state.

Parameters

iteration (int, optional) – if defined the iteration’s state is returned

Returns

job’s or job’s iteration current state

Return type

JobState

str_state(iteration=None)

Return job’s or job’s iteration current state as string.

Parameters

iteration (int, optional) – if defined the iteration’s state is returned

Returns

job’s or job’s iteration current state as string

Return type

JobState

property iteration_states

list of iteration states

Type

list(SubJobState)

set_state(state, iteration=None, err_msg=None)

Set current job’s or job’s iteration state.

Parameters

• state (JobState) – new job’s or job’s iteration state

• iteration (int, optional) – job’s iteration index if the iteration state should be set

• err_msg (str, optional) – the optional error message to record

Returns

the job’s finish state if job’s iteration status change triggered job status change (for example

the last iteration job finished, so the whole job also finished), or None if job’s as a whole still not finished

Return type

JobState

property has_dependencies

true if job has runtime dependencies

Type

bool

append_message(msg)

Record job’s error message.

Parameters

msg (str) – error message to record

queue_pos()

Return current position of a job in scheduling queue

Returns

current position of a job in scheduling queue

Return type

int

set_queue_pos(pos)

Set current position of a job in scheduling queue.

Parameters

pos (int) – current position of a job in scheduling queue

clear_queue_pos()

Reset current position of a job in scheduling queue.

append_runtime(data, iteration)

Record job’s or job’s iteration runtime statistics.

Parameters

• data (dict) – the data to append to job’s or job’s iteration runtime statistics

• iteration (int, optional) – if defined the iteration’s runtime statistics will be updated

• data – the data to append to job’s or job’s iteration runtime statistics

to_dict()

Serialize job’s description to dictionary.

Returns

dictionary with job description

Return type

dict(str)

to_json()

Serialize job description to JSON format.

Returns

JSON of job’s description

class qcg.pilotjob.joblist.JobList

Bases: object

The list of all submited jobs.

_jmap

dictionary with all submited jobs with name as key

Type

dict(str,Job)

Initialize the list.

static parse_jobname(jobname)

Split given name into job name and iteration.

Parameters

jobname (str) – the name to parse

Returns

tuple with job name and iteration, if given job name didn’t contain iteration index, the

second value will be None

Return type

name, iteration

add(job)

Add a new job.

Parameters

job (Job) – job to add to the list

Raises

JobAlreadyExist – when job with given name already exists in list

exist(jobname)

Check if job with given name is in list.

Parameters

jobname (str) – job name to check

Returns

true if job with given name is already in list

Return type

bool

get(jobname)

Return job with given name.

Parameters

jobname (str) – job name

Returns

job from the list or None if not such job has been added.

Return type

Job

jobs()

Return all job names in the list.

Returns

set-like object with job names

remove(jobname)

Remove job with given name from list.

Parameters

jobname (str) – job’s name to remove from list