GuidesChangelogData Inspector Library API Reference
Guides
  1. Command Line Interface
  2. OLP CLI Command reference

pipeline version

pipeline version

The OLP CLI supports the following:

  • create a pipeline version
  • activate a pipeline version
  • upgrade a pipeline version
  • pause a pipeline version
  • resume a pipeline version
  • list a pipeline version
  • show a pipeline version
  • delete a pipeline version
  • cancel a pipeline version
  • deactivate a pipeline version
  • wait for a pipeline version
  • copy a pipeline version
  • get a pipeline version's logging configuration
  • set a pipeline version's logging configuration

pipeline version create

Creates a pipeline version for the specified pipeline.

olp pipeline version create <name> <pipeline ID> <template ID> \
   <pipeline-config.conf> [command options]
olp pipeline version create <name> <pipeline ID> <template ID> ^
   <pipeline-config.conf> [command options]

Required parameters:

  • <name> The name of the version. The length of this parameter must be between 3 and 64 characters.

  • <pipeline ID> The ID of the pipeline.

  • <template ID> The ID of the pipeline template.

  • <pipeline-config.conf> A configuration file with the specified output and input catalogs.

    Typically, the input-catalogs parameter specifies one or more input catalogs for the pipeline. For each input catalog, a fixed identifier is provided together with the catalog's HRN.

    The same catalog can be used for both input and output for stream and batch pipeline versions.

    If the pipeline is in a project scope you can either use a Template in the Project or linked to the Project to create a version. To find pipeline templates available to be linked to a project, use the olp resource link availability create command.

    The pipeline-config.conf file below illustrates how to specify catalogs:

pipeline.config {
      output-catalog { hrn = "hrn:here:data::org:example-output" }
      input-catalogs {
          catalogExample1 { hrn = "hrn:here:data::org:example1" }
          catalogExample2 { hrn = "hrn:here:data::org:example2" }
          catalogExample3 { hrn = "hrn:here:data::org:example3" }
      }
  }

Optional parameters:

  • --supervisor-units <size in units> The size of a supervisor node (1 to 15 units). The pipeline template provides a default value.

  • --supervisor-units-profile <resource profile ID> The ID of the resource profile requested for the supervisor units. The pipeline template provides a default value. To list all the currently available resource profiles, enter the olp pipeline profile list command.

  • --supervisor-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional non-heap memory per supervisor unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05 (but this could be raised on a realm basis).

  • --workers <number of workers> The number of workers. The pipeline template provides a default value.

  • --worker-units <size in units> The size of a worker node (1 to 15 units). The pipeline template provides a default value.

  • --worker-units-profile <resource profile ID> ID of the resource profile requested for the worker units. The pipeline template provides a default value. To list all the currently available resource profiles, enter the olp pipeline profile list command.

  • --worker-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional non-heap memory per worker unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05 (but this could be raised on a realm basis).

  • --garbage-collector-profile <profile ID> ID of the garbage collector profile that is used to configure the garbage collection strategy for the pipeline execution. To list all the currently available garbage collector profiles, enter the olp pipeline environment list command.

    Note::

    If garbage collector profile is not specified during the pipeline version creation, a profile from the appropriate pipeline template is used.

  • --billing-tag <tag,tag1,...> A comma separated tags for cost allocation (up to 6 billing tags), used to group billing records.

  • --runtime-config <key1=value1 key2=value2 ...> A custom map of key-value configuration for the pipeline application. Runtime configuration settings supplied here are added to any default runtime configuration settings from the parent pipeline template. Any runtime configuration settings here override any defaults from the parent pipeline template that are using the same key.

  • --stream-config <key1=value1 key2=value2 ...> A custom map of a key-value configuration for the stream pipeline version. With this parameter, you can pass key-value pairs to specify a custom configuration for stream-2.x.x and newer pipelines. For all supported parameters under this option, see Stream Configuration.

    Note

    The value of the --runtime-config and --stream-config parameters should be quoted with double quotes, like "key1=value1 key2=value2". In case some value contains spaces, it should be quoted with single quotes like "key1='some value1' key2=value2". Take into account that a parameter value with the \n splitter is also acceptable. Example: "key1='some value1'\nkey2=value2\n"

  • --multi-region For a pipeline that requires minimum downtime, use this flag to enable the multi-region setup for the pipeline version so that when the primary region fails, the pipeline version gets automatically transferred to the secondary region. To learn more about the multi-region functionality of pipelines, see Enable Multi-Region Setup for Pipelines

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.

  • --json Displays the pipeline version in JSON format.

  • --quiet Displays the pipeline version ID.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

Deprecated optional parameters:

  • --with-scheduler Sets up a scheduler to run the pipeline version when an input catalog changes. If not set, the pipeline version runs immediately after activation. This parameter is not applicable to a stream pipeline version. Deprecated.

    A new parameter, --schedule, was added to the olp pipeline version activate command to change the mode of execution of a pipeline version.

Note

HERE platform supports "Run As" to allow upgrading pipelines with a different app. The OLP CLI always upgrades a pipeline using the app_id from the credentials file. If you want to upgrade your pipeline with a different app_id, use a different credentials file.

For more information on using credentials and profiles, see Credentials setup.

For more information on pipeline configuration, see Pipeline Configuration.

For more information on additional non-heap memory, see Additional Non-Heap Memory.

Example:

The command below creates a pipeline version using the specified name, pipeline ID, pipeline template ID, and configuration.

olp pipeline version create version-name f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   5c0660a3-0fb4-4f35-bcd0-be6ce25075f6 /user/data/pipeline-config.conf
olp pipeline version create version-name f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   5c0660a3-0fb4-4f35-bcd0-be6ce25075f6 /user/data/pipeline-config.conf

Output:

Pipeline version has been created
ID: d77f288e-2c89-4c94-b4ba-79fbd1e26e79

pipeline version activate

Activates the specified pipeline version and waits for the activate operation to complete. If the operation is successful, the pipeline status becomes either scheduled or running.

olp pipeline version activate <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --input-catalogs <input catalog specs...> Specific versions of input catalogs and execution mode for running a pipeline version job on demand only. This parameter applies to activating a batch pipeline version that needs to be executed with exact versions of input catalogs (defined in the pipeline-config.conf file and type of processing in the run-now mode without a scheduler (that is, with the --schedule parameter set to none). You are recommended to use --input-catalogs for testing and fine-tuning your pipeline version's run configuration locally on specific catalog versions before deploying it to the Here platform Pipelines API. For more information, see Data Processing Library examples that are part of the SDK for Java & Scala archive.

    This parameter accepts one of the following:

    • Path to the pipeline-job.conf file.
    • Input catalog versions and processing type in the catalog-id:processing-type:version[:since-version] format, where since-version is required only for the changes processing type. catalog-id is a fixed identifier used in both pipeline-config.conf and pipeline-job.conf. The processing-type describes what has changed in each input catalog since the last successful run; possible values are: no_changes, changes, reprocess.

    If --input-catalogs is not passed, the latest input catalog versions are automatically identified, and the reprocess processing type is applied.

  • --schedule <data-change|time:<cron-expression>|none> The execution mode of a batch pipeline version. The data-change value configures the pipeline version to run based on input catalog changes. The value none configures the pipeline version to run immediately. If this parameter is not used, the current mode of execution is used for activation. Check the current mode using the pipeline version show command. This parameter is not applicable to a stream pipeline version. The value time:<cron-expression> configures the pipeline version to run based on the schedule defined by a cron expression in the UNIX cron format.

  • --processing <changes|force> The processing type for the time-based schedule. _ Use changes (the default value) to first check for changes in the input catalog before running on the cron schedule. If there are changes in catalog versions, the system submits the job with the catalog versions. _ Use force to run the pipeline version on the given cron schedule without checking for changes in the input catalogs. If --input-catalogs is supplied, the system submits the job with provided catalog versions from the input with every run. If --input-catalogs is not supplied, the system submits the job without catalog versions.

    Note

    The processing parameter is only applicable for time-based scheduling, that is with --schedule time:cron-expression.

  • --high-availability The flag that enables a stream pipeline job to automatically recover from a failed Flink JobManager by using a secondary JobManager when the primary JobManager fails. This feature introduces additional cost for the extra resources. By default, the parameter value is false and a single JobManager is used for the stream pipeline job.

  • --snapshot-id The snapshot ID from a streaming pipeline. Resume a new job from the specified snapshot. For details, see the olp pipeline snapshot command.

    Note

    The --snapshot-id parameter is only applicable to streaming pipelines using runtime environment stream-5.0 or newer.

  • --with-latest-checkpoint The flag that starts a streaming pipeline from the latest checkpoint. If no checkpoint is found, the pipeline version activate operation fails.

    Note

    The --with-latest-checkpoint parameter is only applicable to streaming pipelines using runtime environment stream-5.0 or newer.

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.

  • --json Displays the activated pipeline version contents in JSON format.

  • --quiet Displays the pipeline version state.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

Warning

Backward compatibility

If a batch pipeline version is created using the deprecated --with-scheduler parameter and then activated without the --schedule parameter, it continues to work as expected. However, if the --schedule none option is used during the activation, the pipeline version gets reconfigured to run immediately (on demand). To change the mode back to running based on input catalog changes, use --schedule data-change for the next activation or use --schedule time:<cron-expression> to run the pipeline version on a time-based schedule.

Note

The HERE platform supports "Run As" to allow upgrading pipelines with a different app. The OLP CLI always upgrades a pipeline using the app_id from the credentials file. If you want to upgrade your pipeline with a different app_id, use a different credentials file.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below activates the specified pipeline version.

olp pipeline version activate f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been activated
Current state: scheduled

If the pipeline version was created without the --with-scheduler parameter, the pipeline version starts to run immediately, the pipeline log URL is returned, and the link to the pipeline logs in Splunk is displayed.

If the pipeline version was created with the --with-scheduler parameter, the pipeline version runs when the scheduler is triggered upon input catalog changes.

Using pipeline-job.conf with --input-catalogs

An example of a pipeline-job.conf file is given below:

pipeline.job.catalog-versions {
    output-catalog { base-version = 1 }
    input-catalogs {
        catalogExample1 {
            processing-type = "no_changes"
            version = 1
        }
        catalogExample2 {
            processing-type = "no_changes"
            version = 1
        }
    }
}

For more information, see Interfacing with the Pipeline Process.

pipeline version upgrade

Upgrades the specified pipeline version to a different pipeline version and waits for the operation to complete.

olp pipeline version upgrade <pipeline ID> --to <pipeline version ID to upgrade to> \
   --from <pipeline version ID to upgrade from> [command option]
olp pipeline version upgrade <pipeline ID> --to <pipeline version ID to upgrade to> ^
   --from <pipeline version ID to upgrade from> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.

  • --to <pipeline version ID to upgrade to> The ID of the pipeline version to upgrade to.

  • --from <pipeline version ID to upgrade from> The ID of the pipeline version to upgrade from.

    Note

    The from pipeline version should be in the running or paused state when the command is executed.

    If the from batch pipeline version has been activated to run in On-demand (Run Now) mode, the upgrade operation will fail - only batch pipeline versions activated in Scheduled mode can be upgraded from.

Optional parameters:

  • --high-availability A flag that enables a stream pipeline job to automatically recover from a failed Flink JobManager by using a secondary JobManager when the primary JobManager fails. This feature introduces additional cost for the extra resources. By default, the parameter is false and a single JobManager is used for the stream pipeline job.

  • --schedule <data-change|time:<cron-expression>|none> The execution mode of a batch pipeline version. The data-change value configures the pipeline version to run based on input catalog changes. The time:<cron-expression> value configures the pipeline version to run based on the schedule defined by a cron expression in the UNIX cron format. The value none configures the pipeline version to run immediately. If this parameter is not used, the current mode of execution is used for activation. Check the current mode using the pipeline version show command. This parameter is not applicable to a stream pipeline version.

  • --processing <changes|force> The processing type for the time-based schedule. _ Use changes (the default value) to first check for changes in the input catalog before running on the cron schedule. If there are changes in catalog versions, the system submits the job with the catalog versions. _ Use force to run the pipeline version on the given cron schedule without checking for changes in the input catalogs. If --input-catalogs is supplied, the system submits the job with provided catalog versions from the input with every run. If --input-catalogs is not supplied, the system submits the job without catalog versions.

    Note

    The processing parameter is only applicable for time-based scheduling, that is with --schedule time:cron-expression.

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.

  • --json Displays the upgraded pipeline version contents in JSON format.

  • --quiet Displays the pipeline version state.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

Note

The HERE platform supports "Run As" to allow upgrading pipelines with a different app. The OLP CLI always upgrades a pipeline using the app_id from the credentials file. If you want to upgrade your pipeline with a different app_id, use a different credentials file.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below upgrades the pipeline version from the specified version ID to a new version ID.

olp pipeline version upgrade f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 --to a21y693s-5f83-1h57-a0rr-28lrc5f10g93 \
   --from d77f288e-2c89-4c94-b4ba-79fbd1e26e79
olp pipeline version upgrade f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 --to a21y693s-5f83-1h57-a0rr-28lrc5f10g93 ^
   --from d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been upgraded
State: running

pipeline version pause

Pauses the specified pipeline version that was previously in the running state. The pause operation follows different patterns for stream and batch pipeline versions. For more information, see Running a Pipeline.

olp pipeline version pause <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the paused pipeline version contents in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below pauses the specified pipeline version.

olp pipeline version pause f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been paused
State: paused

pipeline version resume

Resumes the specified paused pipeline version. A stream pipeline version resumes from where it was paused, if possible. A batch pipeline version resumes if and when a new job is available for it to process.

olp pipeline version resume <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --high-availability A flag that enables a stream pipeline job to automatically recover from a failed Flink JobManager by using a secondary JobManager when the primary JobManager fails. This feature introduces additional cost for the extra resources. By default, the parameter is false and a single JobManager is used for the stream pipeline job.
  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the resumed pipeline version contents in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below resumes the specified pipeline version.

olp pipeline version resume f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been resumed
State: scheduled

pipeline version list

Lists all pipeline versions belonging to a specified pipeline ID.

olp pipeline version list <pipeline ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the list of pipeline versions in JSON format.
  • --quiet Displays pipeline version IDs, each on a new line.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below lists all the versions belonging to the specified pipeline.

olp pipeline version list f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 --json

Output:

{"pipelineVersions": [{
    "multiRegionEnabled": false,
    "outputCatalogHRN": "hrn:here:data::org:test-catalog",
    "billingTag": null,
    "created": "2018-03-01T15:21:59.145Z",
    "versionNumber": 1,
    "clusterConfiguration": {
        "workerResourceProfileId": "HS1B",
        "supervisorResourceProfileId": "HS1B",
        "workerAdditionalOverheadMemory": 0,
        "gcProfile": "parallelgc",
        "supervisorUnits": 1,
        "workerUnits": 1,
        "workers": 1,
        "supervisorAdditionalOverheadMemory": 0
    },
    "name": "Version 1",
    "customRuntimeConfiguration": "",
    "schedulerConfiguration": {
        "catalogTriggerOptions": null,
        "timeTriggerOptions": {
            "cronJobExpression": "0 1 * * *",
            "processingType": "changes",
        }
    },
    "streamConfiguration": {"taskmanager.numberOfTaskSlots": 2},
    "id": "d77f288e-2c89-4c94-b4ba-79fbd1e26e79",
    "state": "ready",
    "pipelineTemplateId": "5c0660a3-0fb4-4f35-bcd0-be6ce25075f6",
    "loggingUrl": null,
    "updated": "2018-03-01T15:25:34.591Z",
    "inputCatalogs": {"input":
    "hrn:here:data::org:here-optimized-map-for-location-libraries-na-weu-1"}
}]}

pipeline version show

Shows all the information about the specified pipeline version.

olp pipeline version show <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the list of pipeline versions in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below displays information on the specified pipeline version.

olp pipeline version show f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json
olp pipeline version show f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json

Output:

{
    "multiRegionEnabled": false,
    "outputCatalogHRN": "hrn:here:data::org:test-catalog",
    "billingTag": null,
    "created": "2018-03-01T15:21:59.145Z",
    "versionNumber": 1,
    "clusterConfiguration": {
        "workerResourceProfileId": "HS1B",
        "supervisorResourceProfileId": "HS1B",
        "workerAdditionalOverheadMemory": 0,
        "gcProfile": "parallelgc",
        "supervisorUnits": 1,
        "workerUnits": 1,
        "workers": 1,
        "supervisorAdditionalOverheadMemory": 0
    },
    "name": "v1",
    "highAvailability": false,
    "customRuntimeConfiguration": "",
    "schedulerConfiguration": {
        "catalogTriggerOptions": null,
        "timeTriggerOptions": {
            "cronJobExpression": "0 1 * * *",
            "processingType": "changes",
        }
    },
    "nextJobAttempt": "2019-11-05T01:45:20Z"
    "streamConfiguration": {"taskmanager.numberOfTaskSlots": 2},
    "id": "d77f288e-2c89-4c94-b4ba-79fbd1e26e79",
    "state": "ready",
    "pipelineTemplateId": "5c0660a3-0fb4-4f35-bcd0-be6ce25075f6",
    "loggingUrl": null,
    "updated": "2018-03-01T15:25:34.591Z",
    "inputCatalogs": {"input":
    "hrn:here:data::org:here-optimized-map-for-location-libraries-na-weu-1"}
}

In the verbose output, the schedule field has replaced the schedulerConfiguration field, and human-readable values such as none, data-change, and <cron-expression> are now reported. The next run attempt field shows the date and time when the next job will be run for pipeline versions with the time-based schedule configuration.

Note

High availability

In the verbose output of the olp pipeline version show command, the high availability status is shown as true if the pipeline version has been activated, resumed, or upgraded with the --high-availability flag and has a job in states running or starting.

pipeline version delete

Deletes the specified pipeline version if that pipeline version is in the ready state. This means that to delete a running or paused pipeline, you need to cancel it first.

olp pipeline version delete <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --force Cancels and deactivates a pipeline version, if needed, before deletion.
  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --quiet Displays empty output with no additional information.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below deletes the specified pipeline version.

olp pipeline version delete f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version d77f288e-2c89-4c94-b4ba-79fbd1e26e79 has been deleted

pipeline version cancel

Cancels the specified running pipeline version and any future jobs scheduled for that pipeline version.

olp pipeline version cancel <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the pipeline version contents after cancellation in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below cancels the specified pipeline version.

olp pipeline version cancel f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been canceled
State: ready

pipeline version deactivate

Deactivates the specified activated pipeline version and waits for the operation to complete.

olp pipeline version deactivate <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the deactivated pipeline version contents in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below deactivates the specified pipeline version.

olp pipeline version deactivate f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been deactivated
State: ready

pipeline version wait

Waits for the pipeline version to reach the specified job state. If the pipeline version is in the ready or paused state, the command does not wait for the timeout. Instead, it checks that the job state is reached.

olp pipeline version wait <pipeline ID> <pipeline version ID> \
   --job-state=completed [command options]
olp pipeline version wait <pipeline ID> <pipeline version ID> ^
   --job-state=completed [command options]

Once the job state is reached, the command displays the pipeline job information. Otherwise, an error message is displayed.

Optional parameters:

  • --job-state <running|completed|failed|canceled> The job state for the pipeline version. If not specified, the default is completed.
  • --timeout <duration in seconds> The timeout duration in seconds. If not specified, the default is 3600.
  • --no-error Set this flag to override the state not reached timeout error.
  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the pipeline version job in JSON format if the command succeeds.
  • --quiet Displays empty output with no additional information.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information, see Pipeline States.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below waits for a pipeline job to complete.

olp pipeline version wait f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 \
   --job-state=completed --json
olp pipeline version wait f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 ^
   --job-state=completed --json

Output:

{
  "catalogVersions": null,
  "created": "2018-03-01T15:23:04.618Z",
  "id": "e8e31070-4f4b-4e4b-a8e0-40425844cb75",
  "state": "failed",
  "updated": "2018-03-01T15:23:32.497Z",
  "loggingUrl": null
}

pipeline version copy

Copies a specified pipeline version and its logging configuration. The copied version is in the ready state and can be later activated manually. Use this command to speed up the process of configuring your pipelines.

olp pipeline version copy <pipeline ID> <pipeline version ID> [command options]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --jar <file path> A fat JAR file to upload, including its path on a local file system. If provided, the pipeline version is created with this template. For more information on the JAR file contents and build instructions, have a look at our tutorial on Running a Stream Application with Pipeline Service.

  • --pipeline-config <file path> A configuration file with the specified output and input catalogs.

  • --with-scheduler Sets up a scheduler to run the pipeline version when an input catalog changes. If not set, the pipeline version runs immediately after activation. This parameter does not apply for a stream environment. Deprecated.

    A new parameter, --schedule, was added to the olp pipeline version activate command to change the mode of execution of the pipeline version.

  • --supervisor-units <size in units> The size of a supervisor node (1 to 15 units). If not specified, the value is taken from the original pipeline version.

  • --supervisor-units-profile <resource profile ID> The ID of the resource profile requested for the supervisor units. If not specified, the value is taken from the original pipeline version. To list all the currently available resource profiles, enter the olp pipeline profile list command.

  • --supervisor-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional non-heap memory per supervisor unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05 (but this could be raised on a realm basis).

  • --workers <number of workers> The number of workers. If not specified, the value is taken from the original pipeline version.

  • --worker-units <size in units> The size of a worker node (1 to 15 units). If not specified, the value is taken from the original pipeline version.

  • --worker-units-profile <resource profile ID> ID of the resource profile requested for the worker units. If not specified, the value is taken from the original pipeline version. To list all the currently available resource profiles, enter the olp pipeline profile list command.

  • --worker-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional non-heap memory per worker unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05 (but this could be raised on a realm basis).

  • --garbage-collector-profile <profile ID> ID of the garbage collector profile that is used to configure the garbage collection strategy for the pipeline execution. To list all the currently available garbage collector profiles, enter the olp pipeline environment list command.

    Note::

    If garbage collector profile is not specified during the pipeline version creation, a profile from the appropriate pipeline template is used.

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.

  • --json Displays the pipeline version job in JSON format if the command runs successfully.

  • --quiet Displays the pipeline version ID.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

For more information on pipeline configuration, see Interfacing with the Pipeline Process.

For more information on additional non-heap memory, see Additional Non-Heap Memory.

Example:

The command below copies a pipeline version and customizes the copied pipeline version with a new configuration file and fat JAR file.

olp pipeline version copy f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 \
   --pipeline-config example-pipeline-configuration.conf --jar example.jar
olp pipeline version copy f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 ^
   --pipeline-config example-pipeline-configuration.conf --jar example.jar

Output:

Pipeline version has been copied
ID: 143ww7ux-101d-46c4-8bdd-f8919067bba4

pipeline version log level get

Shows the current logging configuration for a specified pipeline version.

olp pipeline version log level get <pipeline ID> <pipeline version ID> [command parameters]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the logging configuration in JSON format.
  • --quiet Displays the logging levels for the specified pipeline version separated by space.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using logs, see Logs, Monitoring, and Alerts User Guide.

For more information on using credentials and profiles, see Credentials setup.

Example:

olp pipeline version log level get \
   f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json
olp pipeline version log level get ^
   f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json

Output:

{"configuration": {"loggers": {
    "logger": [
        {
            "level": "info",
            "name": "com.example.someLoggingClass"
        },
        {
            "level": "info",
            "name": "com.example.examplepkg.otherClass"
        }
    ],
    "root": {"level": "error"}
}}}

pipeline version log level set

Sets the current logging configuration for a specified pipeline version.

olp pipeline version log level set <pipeline-id> <pipeline-version-id> [command parameters]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.
  • --log4j-properties <property file> Set the logging levels in the log4j format from the property file (see below for a sample property file content).
  • --root <log level> Set the root level to one of the four values: debug, info, warn, or error. The root logger can be configured through the command line. However, the use of this parameter doesn't overwrite the existing configuration set for each logger.

Warning

Setting the log level

To set the log level, use either the --log4j-properties or the root parameter.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the logging configuration in JSON format.
  • --quiet Displays the logging levels for the specified pipeline versions, separated by space.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using logs, see Logs, Monitoring and Alerts User Guide.

For more information on using credentials and profiles, see Credentials setup.

Example:

olp pipeline version log level set f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --root DEBUG --json
olp pipeline version log level set f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --root DEBUG --json

Output:

{
  "configuration": {
    "loggers": {
      "root": { "level": "debug" }
    }
  }
}

You can configure the root and separate loggers by providing log4j 1.x properties in the property file; only rootLogger and logger properties are consumed.

Example of a log4j property file:

# initialize root logger with level ERROR
log4j.rootLogger=ERROR
# set the log level for these components
log4j.logger.com.example.someLoggingClass=INFO
log4j.logger.com.example.examplepkg.otherClass=INFO

For more information on log4j, see the log4j documentation.

Updated 21 days ago