Executor API

Beta

Executor API is an alternative, simplified programming interface for QCG-PilotJob. In some aspects it mimics an interface of concurrent.futures.Executor and may therefore be appealing to many Python programmers. However, since this interface is still under development, it is dedicated mostly for less-demanding use-cases.

Executor API is based on the basic API of QCG-PilotJob and therefore it inherits core elements from that API. On the other hand, in order to support definition of the common execution scenarios, many elements of basic API have been hidden behind simplified interface.

Installation

Executor API can be installed from PyPi, with the following command:

$ pip install qcg-pilotjob-executor-api

Usage

Before we present more details about the usage of Executor API, let’s outline a minimal working example:

from qcg.pilotjob.executor_api.qcgpj_executor import QCGPJExecutor
from qcg.pilotjob.executor_api.templates.basic_template import BasicTemplate

with QCGPJExecutor() as e:
    f = e.submit(BasicTemplate.template, name='tj', exec='date')
    f.result()

This example shows how Executor API can be used to run specific command, here date, within a QCG-PilotJob task.

The interesting part starts on the 4th line. Here we create QCGPJExecutor, which is an entry point to QCG-PilotJob. Actually, behind the scenes QCGPJExecutor initialises the QCG-PilotJob manager service and it plays a role of a proxy to its methods.

Once created, QCGPJExecutor allows us to submit tasks for the execution within QCG-PilotJob. An example invocation of the submit method is shown on the 5th line. The first and the most interesting argument to this method is template. The template is actually a Callable that returns a tuple consisting of string and dictionary. The string need to be a QCG-PilotJob submit request description written in a JSON format with optional placeholders for substitution of specific parameters, while the dictionary may be used to set default values for placeholders. The next parameters of the method are optional and dependent on the selected template - their role is to provide values for the actual substitution of placeholders.

In the example above we use a predefined template called BasicTemplate.template, which requires only two parameters to be provided, namely name and exec.

The submit method returns a QCGPJFuture object, which provides methods associated with the execution of submission. For instance, the invocation f.result() in the example above, blocks processing until the task is not completed and then returns the status of its execution.

QCGPJExecutor

QCGPJExecutor is an approximate implementation of the concurrent.futures.Executor interface, but instead of execution of functions using threads or multiprocessing module like it takes place in case of python build-in executors, here we execute QCG-PilotJob’s tasks.

Technically, QCGPJExecutor is a kind of proxy over the QCG-PilotJob manager and at the expense of some flexibility of the covered service, it provides simpler interface. QCGPJExecutor’s constructor can be invoked without any parameters and then it is started with default settings.

However, in order to enable easy configuration of the commonly changed settings, several optional parameters are provided. One of such parameters is resources which may be useful for testing QCG-PilotJob on a local laptop.

QCGPJExecutor implements ContextManager’s methods that allow for its easy usage with the with statements. When the with statement is used, python will automatically take care of releasing QCGPJExecutor’s resources.

When the QCGPJExecutor is constructed outside the with statement, it needs to be released manually, using the shutdown method.

For the full reference of the QCGPJExecutor module see qcg.pilotjob.executor_api.qcgpj_executor.

Submission of tasks

The key method offered by QCGPJExecutor is submit. The call of this method adds a new task (or tasks, depending on the usage scenario) to the QCG-PilotJob’s queue to be executed once resources are available and dependencies satisfied. The method takes the following arguments:

1. fn :

   a callable that returns a tuple representing a template. The first element of the tuple should be a string containing a QCG-PilotJob submit request expressed in a JSON format compatible with the QCG-PilotJob’s interface. The string can include placeholders (identifiers preceded by $ symbol) that are the target for substitution. The second element of a tuple is dictionary which may be used to assign default values for substitution of selected placeholders.

2. *args :

   a set of dicts which contain parameters that will be used to substitute placeholders defined in the template.

3. **kwargs :

   a set of keyword arguments that will be used to substitute placeholders defined in the template.

Note: In the process of substitution **kwargs overwrite *args and *args overwrite defaults

Example template

In order to understand how to use or create templates, possibly the best option is to look at the example. BasicTemplate class, which is delivered with the QCG-PilotJob Executor API, provides a predefined template method that was already used in the example above. It is a simple example, but can give a good overview.

class BasicTemplate(QCGPJTemplate):
    @staticmethod
    def template() -> Tuple[str, Dict[str, Any]]:
        template = """
        {
            'name': '${name}',
            'execution': {
                'exec': '${exec}',
                'args': ${args},
                'stdout': '${stdout}',
                'stderr': '${stderr}'
            }
        }
         """

        defaults = {
            'args': [],
            'stdout': 'stdout',
            'stderr': 'stderr'
        }

        return template, defaults

Here, accordingly with the expectations, the function returns template and defaults. The template is a JSON dictionary representing a QCG-PilotJob submit request. What is important, it includes a set of ${} placeholders. These placeholders may be substituted by the parameters provided to the submit method. For some of the placeholders, default values are already predefined in a defaults dictionary, and these parameters don’t need to be substituted if there is no concrete reason for this. The rest of placeholders, namely {name} and {exec}, don’t have default values and therefore they need to be substituted by parameters provided to the submit.

Let’s see how example invocations of the submit method for this template can look like:

e.submit(BasicTemplate.template, name='tj', exec='date')
e.submit(BasicTemplate.template, name='tj', exec='sleep', args=['10'])

QCGPJFuture

The submit method returns QCGPJFuture object, which plays a role of a handler for the submission. Thus, using the returned QCGPJFuture object it is possible to make queries to check if the submitted task has been finished, with the done method, or request the cancellation of an execution with the cancel method. As it was presented in the attached example, it is also possible to invoke blocking wait until the task is finished with the result method. For the full reference of methods provided by QCGPJFuture see qcg.pilotjob.executor_api.qcgpj_future.