pipeline template

The OLP CLI supports the following:

• list pipeline templates
• create pipeline templates
• show pipeline templates
• delete pipeline templates

pipeline template list

Lists all pipeline templates to which you have access.

olp pipeline template list [filter] [command options]

Optional parameters:

• [filter] Freeform text for filtering the pipeline template list: the filter checks if the name or the description field contains the filter string.
• --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
• --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
• --json Displays the command result in JSON format.
• --quiet Displays pipeline template IDs, each on a new line.
• --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below lists all the pipeline templates to which you have access and whose name or description field contains the word spark.

olp pipeline template list "spark" --json

Output:

{
    "pipelineTemplates": [
        {
            "created": "2018-03-01T15:16:53.796Z",
            "groupId": "GROUP-9479863e-a13b-4d35-9eb1-5a054669046e",
            "defaultClusterConfiguration": {
                "workerResourceProfileId": "HS1B",
                "supervisorResourceProfileId": "HS1B",
                "workerAdditionalOverheadMemory": 0,
                "gcProfile": "parallelgc",
                "supervisorUnits": 1,
                "workerUnits": 1,
                "workers": 1,
                "supervisorAdditionalOverheadMemory": 0
            },
            "name": "sparktestcompiler",
            "packageId": "68d723f6-2ae7-40e4-8c24-61512d511852",
            "entryPointClassName": "com.example.Main",
            "description": "",
            "id": "5c0660a3-0fb4-4f35-bcd0-be6ce25075f6",
            "state": "created",
            "defaultRuntimeConfiguration": "",
            "updated": "2018-03-01T15:16:53.796Z",
            "runtimeEnvironment": "batch-5.0"
        }
    ]
}

pipeline template create

Creates a pipeline template.

olp pipeline template create <name> <runtime environment> <JAR file> \
 <class name> <group ID> --input-catalog-ids catalogExample1 [command options]

olp pipeline template create <name> <runtime environment> <JAR file> ^
<class name> <group ID> --input-catalog-ids catalogExample1 [command options]

Required parameters:

• <name> The name for a pipeline template. The length of this parameter must be between 3 and 64 characters.

• <runtime environment> A distributed processing framework used to execute the pipeline template with the possible runtime environment ID. To list all the currently enabled runtime environments, enter the olp pipeline environment list command.

• <JAR file> A fat JAR file, including the path on a local file system, to upload.

• <class name> The name of the pipeline template main class.

• Either of the following:

  • <group ID> The ID of the group whose members are allowed to access the pipeline.
  • --scope <project HRN> The HRN of the project whose members are allowed to access the pipeline.

• --input-catalog-ids <fixed catalog identifiers...> Fixed identifiers of the input catalogs expected by the pipeline (separated by space). This list should match the catalog identifiers used in the pipeline-config.conf file.

  Note

  --input-catalog-ids may also accept the path to the pipeline-config.conf configuration file that contains fixed identifiers of catalogs. This file is used when a pipeline version is created.

  For more information on pipeline-config.conf, see Pipeline Configuration.

Optional parameters:

• --description <template description> A description of the pipeline template. The length of this parameter must be between 0 and 512 characters.

• --supervisor-units <size in units> The size of a supervisor node (1 to 15 units). If not specified, the default value is 1.

• --supervisor-units-profile <resource profile ID> The ID of the resource profile requested for the supervisor units. If not specified, the Default profile is used. To list all the currently available resource profiles, enter the olp pipeline profile list command.

• --supervisor-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional non-heap memory per supervisor unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05 (but this could be raised on a realm basis).

• --workers <number of workers> The default number of workers. If not specified, the default value is 1.

• --worker-units <size in units> The default size of a worker node (1 to 15 units). If not specified, the default value is 1.

• --worker-units-profile <resource profile ID> ID of the resource profile requested for the worker units. If not specified, the Default profile is used. To list all the currently available resource profiles, enter the olp pipeline profile list command.

• --worker-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional non-heap memory per worker unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05 (but this could be raised on a realm basis).

• --garbage-collector-profile <profile ID> ID of the garbage collector profile that is used to configure the garbage collection strategy for the pipeline execution. To list all the currently available garbage collector profiles, enter the olp pipeline environment list command.

  Note::

  If garbage collector profile is not specified, the Parallel GC profile is used for batch pipeline templates and the G1 GC profile is used for stream pipeline templates.

• --default-runtime-config <key1=value1 key2=value2 ...> A map of default configuration values for the pipeline application. The maximum key name size is 256, and the maximum value size is 1024. Runtime configuration can be overridden while creating a pipeline version. If a pipeline version has its own runtime configuration values, they are added to any defaults available from the parent pipeline template.

  Note

  The value of the --default-runtime-config parameter should be quoted with double quotes, like "key1=value1 key2=value2". In case some value contains spaces, it should be quoted with single quotes like "key1='some value1' key2=value2". Take into account that a parameter value with the \n splitter is also acceptable. Example: "key1='some value1'\nkey2=value2\n"

• --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.

• --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.

• --json Displays the created pipeline template contents in JSON format.

• --quiet Displays the pipeline template ID.

For more information on using credentials and profiles, see Credentials setup.

For more information on pipeline configuration, see Pipeline Configuration.

For more information on additional non-heap memory, see Additional Non-Heap Memory.

Example:

The command below creates a pipeline template with the specified name, runtime environment, package, main class, project that the pipeline belongs to, and input catalog IDs that are expected in the pipeline version configuration.

olp pipeline template create "example-pipeline-template" batch-5.0 \
example-pipeline.jar com.example.Main \
--input-catalog-ids optimized-map sensor-data \
--scope hrn:here:authorization::org:project/project-example

olp pipeline template create "example-pipeline-template" batch-5.0 ^
example-pipeline.jar com.example.Main ^
--input-catalog-ids optimized-map sensor-data ^
--scope hrn:here:authorization::org:project/project-example

Output:

Pipeline template has been created
ID: 5c0660a3-0fb4-4f35-bcd0-be6ce25075f6

pipeline template show

Shows all the information about the specified pipeline template.

olp pipeline template show <pipeline template ID> [command option]

Required parameters:

• <pipeline template ID> The ID of the pipeline template to show.

Optional parameters:

• --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
• --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
• --json Displays the template information in JSON format.
• --quiet Displays the pipeline template ID.
• --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

Example:

olp pipeline template show 5c0660a3-0fb4-4f35-bcd0-be6ce25075f6 --json

Output:

The example below gets all the information about the specified pipeline template.

{
    "created": "2018-03-01T15:16:53.796Z",
    "groupId": "GROUP-9479863e-a13b-4d35-9eb1-5a054669046e",
    "hrn": "hrn:here:pipeline-template::org:5c0660a3-0fb4-4f35-bcd0-be6ce25075f6",
    "defaultClusterConfiguration": {
        "workerResourceProfileId": "HS1B",
        "supervisorResourceProfileId": "HS1B",
        "workerAdditionalOverheadMemory": 0,
        "gcProfile": "parallelgc",
        "supervisorUnits": 1,
        "workerUnits": 1,
        "workers": 1,
        "supervisorAdditionalOverheadMemory": 0
    },
    "name": "sparktestcompiler",
    "packageId": "68d723f6-2ae7-40e4-8c24-61512d511852",
    "entryPointClassName": "com.example.Main",
    "description": "",
    "id": "5c0660a3-0fb4-4f35-bcd0-be6ce25075f6",
    "state": "created",
    "defaultRuntimeConfiguration": "",
    "updated": "2018-03-01T15:16:53.796Z",
    "runtimeEnvironment": "batch-5.0"
}

pipeline template delete

Deletes the specified pipeline template. The pipeline template that you want to delete cannot be in use. This means that there are no pipeline versions defined with that pipeline template.

olp pipeline template delete <pipeline template ID> [command option]

Warning

Pipeline template is removed completely

This command removes the pipeline template completely, and there is no way to restore it.

Required parameters:

• <pipeline template ID> The ID of the pipeline template that should be deleted.

Optional parameters:

• --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
• --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
• --quiet Displays empty output with no additional information.
• --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below deletes the specified pipeline template.

olp pipeline template delete 5c0660a3-0fb4-4f35-bcd0-be6ce25075f6

Output:

Pipeline template 5c0660a3-0fb4-4f35-bcd0-be6ce25075f6 has been deleted