API Documentation

Comprehensive reference material for most of the public API exposed by Cromwell-tools:

cromwell_tools.cromwell_api

TODO: add some module docs TODO: once switched to support only Py3.7+, replace all ‘cls’ type annotations with the actual Types, rather than using the strings. This in Py3.6(-) is limited by the lack of Postponed Evaluation of Annotations, see: https://www.python.org/dev/peps/pep-0563/

class cromwell_tools.cromwell_api.CromwellAPI

Contains a set of classmethods that implement interfaces to cromwell REST API endpoints.

classmethod abort(uuid: str, auth: cromwell_tools.cromwell_auth.CromwellAuth, raise_for_status: bool = False) → requests.models.Response

Request Cromwell to abort a running workflow by UUID.

Parameters:

• uuid – A Cromwell workflow UUID, which is the workflow identifier.
• auth – The authentication class holding headers or auth information to a Cromwell server.
• raise_for_status – Whether to check and raise for status based on the response.

Raises:

requests.exceptions.HTTPError – This will be raised when raise_for_status is True and Cromwell returns a response that satisfies 400 <= response.status_code < 600.

Returns:

HTTP response from Cromwell.

classmethod health(auth: cromwell_tools.cromwell_auth.CromwellAuth, raise_for_status: bool = False) → requests.models.Response

Return the current health status of any monitored subsystems of the Cromwell Server.

Parameters:

• auth – authentication class holding headers or auth information to a Cromwell server.
• raise_for_status – Whether to check and raise for status based on the response.

Raises:

requests.exceptions.HTTPError – This will be raised when raise_for_status is True and Cromwell returns a response that satisfies 400 <= response.status_code < 600.

Returns:

HTTP response from Cromwell.

classmethod metadata(uuid: str, auth: cromwell_tools.cromwell_auth.CromwellAuth, includeKey: Union[List[str], str] = None, excludeKey: Union[List[str], str] = None, expandSubWorkflows: bool = False, raise_for_status: bool = False) → requests.models.Response

Retrieve the workflow and call-level metadata for a specified workflow by UUID.

Parameters:

• uuid – A Cromwell workflow UUID, which is the workflow identifier.
• auth – The authentication class holding headers or auth information to a Cromwell server.
• includeKey – When specified key(s) to include from the metadata. Matches any key starting with the value. May not be used with excludeKey.
• excludeKey – When specified key(s) to exclude from the metadata. Matches any key starting with the value. May not be used with includeKey.
• expandSubWorkflows – When true, metadata for sub workflows will be fetched and inserted automatically in the metadata response.
• raise_for_status – Whether to check and raise for status based on the response.

Raises:

requests.exceptions.HTTPError – This will be raised when raise_for_status is True and Cromwell returns a response that satisfies 400 <= response.status_code < 600.

Returns:

HTTP response from Cromwell.

classmethod patch_labels(uuid: str, labels: Dict[str, str], auth: cromwell_tools.cromwell_auth.CromwellAuth, raise_for_status: bool = False) → requests.models.Response

Add new labels or patch existing labels for an existing workflow.

Parameters:

• uuid – A Cromwell workflow UUID, which is the workflow identifier.
• labels – A dictionary representing the label key-value pairs.
• auth – The authentication class holding headers or auth information to a Cromwell server.
• raise_for_status – Whether to check and raise for status based on the response.

Raises:

requests.exceptions.HTTPError – This will be raised when raise_for_status is True and Cromwell returns a response that satisfies 400 <= response.status_code < 600.

Returns:

HTTP response from Cromwell.

classmethod query(query_dict: Dict[str, Union[str, List[str], Dict[str, str], bool]], auth: cromwell_tools.cromwell_auth.CromwellAuth, raise_for_status: bool = False) → requests.models.Response

Query for workflows.

TODO: Given that Cromwell-as-a-Service blocks a set of features that are available in Cromwell, e.g. ‘labelor’, for security concerns, the first iteration of this API doesn’t come up with the advanced query keys of the Cromwell except a set of necessary ones. However, we need to implement this for completeness and keep an eye on the compatibility between CaaS and Cromwell.

All of the query keys will be used in an OR manner, except the keys within labels, which are defined in an AND relation. For instance, [{‘status’: ‘Succeeded’}, {‘status’: ‘Failed’}] will give you all of the workflows that in either Succeeded or Failed statuses.

Parameters:

• query_dict – A dictionary representing the query key-value paris. The keys should be accepted by the Cromwell or they will get ignored. The values could be str, list or dict.
• auth – The authentication class holding headers or auth information to a Cromwell server.
• raise_for_status – Whether to check and raise for status based on the response.

Raises:

requests.exceptions.HTTPError – This will be raised when raise_for_status is True and Cromwell returns a response that satisfies 400 <= response.status_code < 600.

Returns:

HTTP response from Cromwell.

classmethod release_hold(uuid: str, auth: cromwell_tools.cromwell_auth.CromwellAuth, raise_for_status: bool = False) → requests.models.Response

Request Cromwell to release the hold on a workflow.

It will switch the status of a workflow from ‘On Hold’ to ‘Submitted’ so it can be picked for running. For a workflow that was not submitted with workflowOnHold = true, Cromwell will throw an error.

Parameters:

• uuid – A Cromwell workflow UUID, which is the workflow identifier. The workflow is expected to have On Hold status.
• auth – The authentication class holding headers or auth information to a Cromwell server.
• raise_for_status – Whether to check and raise for status based on the response.

Raises:

requests.exceptions.HTTPError – This will be raised when raise_for_status is True and Cromwell returns a response that satisfies 400 <= response.status_code < 600.

Returns:

HTTP response from Cromwell.

classmethod status(uuid: str, auth: cromwell_tools.cromwell_auth.CromwellAuth, raise_for_status: bool = False) → requests.models.Response

Retrieves the current state for a workflow by UUID.

Parameters:

• uuid – A Cromwell workflow UUID, which is the workflow identifier.
• auth – The authentication class holding headers or auth information to a Cromwell server.
• raise_for_status – Whether to check and raise for status based on the response.

Raises:

requests.exceptions.HTTPError – This will be raised when raise_for_status is True and Cromwell returns a response that satisfies 400 <= response.status_code < 600.

Returns:

HTTP response from Cromwell.

classmethod submit(auth: cromwell_tools.cromwell_auth.CromwellAuth, wdl_file: Union[str, _io.BytesIO], inputs_files: Union[List[Union[str, _io.BytesIO]], str, _io.BytesIO] = None, options_file: Union[str, _io.BytesIO] = None, dependencies: Union[str, List[str], _io.BytesIO] = None, label_file: Union[str, _io.BytesIO] = None, collection_name: str = None, on_hold: bool = False, validate_labels: bool = False, raise_for_status: bool = False) → requests.models.Response

Submits a workflow to Cromwell.

Parameters:

• auth – authentication class holding auth information to a Cromwell server.
• wdl_file – The workflow source file to submit for execution. Could be either the path to the file (str) or the file content in io.BytesIO.
• inputs_files – The input data in JSON format. Could be either the path to the file (str) or the file content in io.BytesIO. This could also be a list of unlimited input file paths/contents, each of them should have a type of Union[str, io.BytesIO].
• options_file – The Cromwell options file for workflows. Could be either the path to the file (str) or the file content in io.BytesIO.
• dependencies – Workflow dependency files. Could be the path to the zipped file (str) containing dependencies, a list of paths(List[str]) to all dependency files to be zipped or a zipped file in io.BytesIO.
• label_file – A collection of key/value pairs for workflow labels in JSON format, could be either the path to the JSON file (str) or the file content in io.BytesIO.
• collection_name – Collection in SAM that the workflow should belong to, if use CaaS.
• on_hold – Whether to submit the workflow in “On Hold” status.
• validate_labels – If True, validate cromwell labels.
• raise_for_status – Whether to check and raise for status based on the response.

Raises:

requests.exceptions.HTTPError – This will be raised when raise_for_status is True and Cromwell returns a response that satisfies 400 <= response.status_code < 600.

Returns:

HTTP response from Cromwell.

classmethod wait(workflow_ids: List[str], auth: cromwell_tools.cromwell_auth.CromwellAuth, timeout_minutes: int = 120, poll_interval_seconds: int = 30, verbose: bool = True) → None

Wait until cromwell returns successfully for each provided workflow

Given a list of workflow ids, wait until cromwell returns successfully for each status, or one of the workflows fails or is aborted.

Parameters:

• workflow_ids – A list of workflow ids to wait for terminal status.
• timeout_minutes – Maximum number of minutes to wait.
• auth – Authentication class holding headers or auth information to a Cromwell server.
• poll_interval_seconds – Number of seconds between checks for workflow completion.
• verbose – If True, report to stdout when all workflows succeed.

cromwell_tools.cromwell_auth

TODO: add some module docs TODO: once switched to support only Py3.7+, replace all ‘cls’ type annotations with the actual Types, rather than using the strings. This in Py3.6(-) is limited by the lack of Postponed Evaluation of Annotations, see: https://www.python.org/dev/peps/pep-0563/

cromwell_tools.utilities

cromwell_tools.utilities.compose_oauth_options_for_jes_backend_cromwell(auth: cromwell_tools.cromwell_auth.CromwellAuth, cromwell_options_file: _io.BytesIO = None, execution_bucket: str = None) → _io.BytesIO

Append special options that are required by JES(Google Job Execution Service) backend Cromwell.

This helper function will append special options that are required by JES(Google Job Execution Service) backend Cromwell/Cromwell-as-a-Service to the default workflow options. Note: These options only work with Cromwell instances that use the Google Cloud Backend and allow user-service-account authentication.

Parameters:

• auth – authentication class holding auth information to a Cromwell server.
• cromwell_options_file – Optional, contents of the options for a workflow in BytesIO format. if not specified, this function will create an empty option stream and add the necessary keys to it.
• execution_bucket – Optional, the Google CLoud Bucket that Cromwell will use to output execution results and store temporary scripts. If not specified, it will use ‘gs://{google_project}-cromwell-execution/caas-cromwell-executions’ by default.

Returns:

BytesIO object of the updated workflow options with the required auth fields.

Return type:

options_stream

cromwell_tools.utilities.download(url: str) → Union[str, bytes]

Reads the contents located at the url into memory and returns them.

Urls starting with http are fetched with an http request. All others are assumed to be local file paths and read from the local file system.

Parameters:url – The url to the content to be downloaded, or the path to the local file. Returns:Downloaded content in str or bytes format. Raises:TypeError – If the url is not a str type.

cromwell_tools.utilities.download_http(url: str) → Union[str, bytes]

Makes an http request for the contents at the given url and returns the response body.

Parameters:url – The url to the content to be downloaded. Returns:Content returned from the server. Will be str in Python2 and bytes in Python3. Return type:response_str

cromwell_tools.utilities.download_to_map(urls: List[str]) → Dict[str, Any]

Reads contents from each url into memory and returns a map of urls to their contents.

Parameters:urls – A list of urls to the contents to be downloaded. Returns:A dict representing the mapping from url to the downloaded contents in-memory. Return type:url_to_contents

cromwell_tools.utilities.make_zip_in_memory(url_to_contents: Dict[str, Any]) → _io.BytesIO

Given a map of urls and their contents, returns an in-memory zip file containing each file.

For each url, the part after the last slash is used as the file name when writing to the zip archive.

Parameters:url_to_contents – A dict representing the mapping from url to the downloaded contents in-memory. Returns:Zipped files content in bytes. Return type:bytes_buf

cromwell_tools.utilities.prepare_workflow_manifest(wdl_file: Union[str, _io.BytesIO], inputs_files: Union[List[Union[str, _io.BytesIO]], str, _io.BytesIO] = None, options_file: Union[str, _io.BytesIO] = None, dependencies: Union[str, List[str], _io.BytesIO] = None, label_file: Union[str, _io.BytesIO] = None, collection_name: str = None, on_hold: bool = False) → Dict[str, Union[_io.BytesIO, str]]

Prepare the submission manifest for a workflow submission.

Parameters:

• wdl_file – The workflow source file to submit for execution. Could be either the path to the file (str) or the file content in io.BytesIO.
• inputs_files – The input data in JSON format. Could be either the path to the file (str) or the file content in io.BytesIO. This could also be a list of unlimited input file paths/contents, each of them should have a type of Union[str, io.BytesIO].
• options_file – The Cromwell options file for workflows. Could be either the path to the file (str) or the file content in io.BytesIO.
• dependencies – Workflow dependency files. Could be the path to the zipped file (str) containing dependencies, a list of paths(List[str]) to all dependency files to be zipped or a zipped file in io.BytesIO.
• label_file – A collection of key/value pairs for workflow labels in JSON format, could be either the path to the JSON file (str) or the file content in io.BytesIO.
• collection_name – Collection in SAM that the workflow should belong to, if use CaaS.
• on_hold – Whether to submit the workflow in “On Hold” status.

Returns:

A dictionary representing the workflow manifest ready for workflow submission.

Return type:

workflow_manifest

Raises:

ValueError – If a str ing of path to the dependencies is given but not endswith “.zip”.

cromwell_tools.utilities.read_local_file(path: str) → Union[str, bytes]

Reads the file contents and returns them.

Parameters:path – Path to the local file to be loaded. Returns:The loaded content. bytes in Python3 and str in Python2. Return type:contents

cromwell_tools.utilities.validate_cromwell_label(label_object: Union[str, _io.BytesIO, bytes, Dict[str, str]]) → None

Check if the label object is valid for Cromwell.

Note: this function as well as the global variables _CROMWELL_LABEL_LENGTH, _CROMWELL_LABEL_KEY_REGEX and _CROMWELL_LABEL_VALUE_REGEX are implemented based on the Cromwell’s documentation: https://cromwell.readthedocs.io/en/develop/Labels/ and the Cromwell’s code base: https://github.com/broadinstitute/cromwell/blob/master/core/src/main/scala/cromwell/core/labels/Label.scala#L16 Both the docs and the code base of Cromwell could possibly change in the future, please update this checker on demand.

Parameters:label_object – A dictionary or a key-value object string defines a Cromwell label. Raises:ValueError – This validator will raise an exception if the label_object is invalid as a Cromwell label.

cromwell_tools.cli

class cromwell_tools.cli.DefaultHelpParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True)

add_argument(dest, ..., name=value, ...)

add_argument(option_string, option_string, …, name=value, …)

error(message: string)

Prints a usage message incorporating the message to stderr and exits.

If you override this in a subclass, it should not return – it should either exit or raise an exception.