Executor API¶
Beta
Executor API is an alternative, simplified programming interface for QCG-PilotJob.
In some aspects it mimics an interface of concurrent.futures.Executor
and may therefore
be appealing to many Python programmers. However, since this interface is still under development,
it is dedicated mostly for less-demanding use-cases.
Executor API is based on the basic API of QCG-PilotJob and therefore it inherits core elements from that API. On the other hand, in order to support definition of the common execution scenarios, many elements of basic API have been hidden behind simplified interface.
Installation¶
Executor API can be installed from PyPi, with the following command:
$ pip install qcg-pilotjob-executor-api
Usage¶
Before we present more details about the usage of Executor API, let’s outline a minimal working example:
1 2 3 4 5 6 | from qcg.pilotjob.executor_api.qcgpj_executor import QCGPJExecutor from qcg.pilotjob.executor_api.templates.basic_template import BasicTemplate with QCGPJExecutor() as e: f = e.submit(BasicTemplate.template, name='tj', exec='date') f.result() |
This example shows how Executor API can be used to run specific command, here date
, within a QCG-PilotJob task.
The interesting part starts on the 4th line. Here we create QCGPJExecutor
, which is an entry point to
QCG-PilotJob. Actually, behind the scenes QCGPJExecutor
initialises the QCG-PilotJob manager service
and it plays a role of a proxy to its methods.
Once created, QCGPJExecutor
allows us to submit tasks for the execution within QCG-PilotJob.
An example invocation of the submit
method is shown on the 5th line. The first and the most interesting argument
to this method is template. The template is actually a Callable that returns a tuple consisting of
string and dictionary. The string need to be a QCG-PilotJob submit request description written
in a JSON format with optional placeholders for substitution of specific parameters,
while the dictionary may be used to set default values for placeholders.
The next parameters of the method are optional and dependent on the selected template -
their role is to provide values for the actual substitution of placeholders.
In the example above we use a predefined template called BasicTemplate.template
, which requires
only two parameters to be provided, namely name
and exec
.
The submit
method returns a QCGPJFuture
object, which provides methods associated with the execution
of submission. For instance, the invocation f.result()
in the example above, blocks processing until the task
is not completed and then returns the status of its execution.
QCGPJExecutor¶
QCGPJExecutor
is an approximate implementation of the concurrent.futures.Executor
interface, but instead of
execution of functions using threads or multiprocessing module like it takes place in case of python build-in
executors, here we execute QCG-PilotJob’s tasks.
Technically, QCGPJExecutor
is a kind of proxy over the QCG-PilotJob manager and at the expense
of some flexibility of the covered service, it provides simpler interface.
QCGPJExecutor
’s constructor can be invoked without any parameters and then it is started with default settings.
However, in order to enable easy configuration of the commonly changed settings,
several optional parameters are provided. One of such parameters is resources
which may be useful for
testing QCG-PilotJob on a local laptop.
QCGPJExecutor
implements ContextManager’s methods that allow for its easy usage with the with
statements.
When the with
statement is used, python will automatically take care of releasing QCGPJExecutor
’s resources.
When the QCGPJExecutor
is constructed outside the with
statement, it needs to be released manually,
using the shutdown
method.
For the full reference of the QCGPJExecutor
module see qcg.pilotjob.executor_api.qcgpj_executor
.
Submission of tasks¶
The key method offered by QCGPJExecutor is submit
. The call of this method adds a new task (or tasks, depending on
the usage scenario) to the QCG-PilotJob’s queue to be executed once resources are available and dependencies satisfied.
The method takes the following arguments:
fn
:- a callable that returns a tuple representing a template. The first element of the tuple should be a string containing a QCG-PilotJob submit request expressed in a JSON format compatible with the QCG-PilotJob’s interface. The string can include placeholders (identifiers preceded by $ symbol) that are the target for substitution. The second element of a tuple is dictionary which may be used to assign default values for substitution of selected placeholders.
*args
:- a set of dicts which contain parameters that will be used to substitute placeholders defined in the template.
**kwargs
:- a set of keyword arguments that will be used to substitute placeholders defined in the template.
Note: In the process of substitution **kwargs
overwrite *args
and *args
overwrite defaults
Example template¶
In order to understand how to use or create templates, possibly the best option is to look at the example.
BasicTemplate
class, which is delivered with the QCG-PilotJob Executor API, provides a predefined
template method that was already used in the example above. It is a simple example, but can give a good overview.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | class BasicTemplate(QCGPJTemplate): @staticmethod def template() -> Tuple[str, Dict[str, Any]]: template = """ { 'name': '${name}', 'execution': { 'exec': '${exec}', 'args': ${args}, 'stdout': '${stdout}', 'stderr': '${stderr}' } } """ defaults = { 'args': [], 'stdout': 'stdout', 'stderr': 'stderr' } return template, defaults |
Here, accordingly with the expectations, the function returns template
and defaults
.
The template
is a JSON dictionary representing a QCG-PilotJob submit request. What is
important, it includes a set of ${}
placeholders. These placeholders may be substituted by the parameters
provided to the submit
method. For some of the placeholders, default values are already
predefined in a defaults
dictionary, and these parameters don’t need to be substituted
if there is no concrete reason for this. The rest of placeholders, namely {name}
and {exec}
, don’t have
default values and therefore they need to be substituted by parameters provided to the submit
.
Let’s see how example invocations of the submit
method for this template can look like:
e.submit(BasicTemplate.template, name='tj', exec='date')
e.submit(BasicTemplate.template, name='tj', exec='sleep', args=['10'])
QCGPJFuture¶
The submit
method returns QCGPJFuture
object, which plays a role of a handler for the submission.
Thus, using the returned QCGPJFuture
object it is possible to make queries to check if
the submitted task has been finished, with the done
method,
or request the cancellation of an execution with the cancel
method. As it was presented in the
attached example, it is also possible to invoke blocking wait until the task is finished with the result
method.
For the full reference of methods provided by QCGPJFuture
see qcg.pilotjob.executor_api.qcgpj_future
.