GuidesPipeline API ReferenceChangelog
Guides
  1. Command Line Interface
  2. OLP CLI Command reference

pipeline version

pipeline version

The OLP CLI supports the following:

  • create a pipeline version
  • activate a pipeline version
  • upgrade a pipeline version
  • pause a pipeline version
  • resume a pipeline version
  • list a pipeline version
  • show a pipeline version
  • delete a pipeline version
  • cancel a pipeline version
  • deactivate a pipeline version
  • wait for a pipeline version
  • copy a pipeline version
  • get a pipeline version's logging configuration
  • set a pipeline version's logging configuration

pipeline version create

Creates a pipeline version for the specified pipeline.

olp pipeline version create <name> <pipeline ID> <template ID> \
   <pipeline-config.conf> [command options]
olp pipeline version create <name> <pipeline ID> <template ID> ^
   <pipeline-config.conf> [command options]

Required parameters:

  • <name> The name of the version. The length of this parameter must be between
    3 and 64 characters.

  • <pipeline ID> The ID of the pipeline.

  • <template ID> The ID of the pipeline template.

  • <pipeline-config.conf> A configuration file with the specified output
    and input catalogs.

    Typically, the input-catalogs parameter specifies one or more input catalogs
    for the pipeline. For each input catalog, a fixed identifier is provided
    together with the catalog's HRN.

    The same catalog can be used for both input and output for stream and batch pipeline versions.

    If the pipeline is in a project scope you can either use a Template in the Project or
    linked to the Project to create a version. To find pipeline templates available to be
    linked to a project, use the olp resource link availability create command.

    The pipeline-config.conf
    file below illustrates how to specify catalogs:

pipeline.config {
      output-catalog { hrn = "hrn:here:data::org:example-output" }
      input-catalogs {
          catalogExample1 { hrn = "hrn:here:data::org:example1" }
          catalogExample2 { hrn = "hrn:here:data::org:example2" }
          catalogExample3 { hrn = "hrn:here:data::org:example3" }
      }
  }

Optional parameters:

  • --supervisor-units <size in units> The size of a supervisor node
    (1 to 15 units). The pipeline template provides a default value.

  • --supervisor-units-profile <resource profile ID> The ID of the resource profile requested for the supervisor units.
    The pipeline template provides a default value. To list all the currently available resource
    profiles, enter the olp pipeline profile list
    command.

  • --supervisor-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional
    non-heap memory per supervisor unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05
    (but this could be raised on a realm basis).

  • --workers <number of workers> The number of workers. The pipeline template
    provides a default value.

  • --worker-units <size in units> The size of a worker node (1 to 15 units).
    The pipeline template provides a default value.

  • --worker-units-profile <resource profile ID> ID of the resource profile requested for the worker units.
    The pipeline template provides a default value. To list all the currently available resource
    profiles, enter the olp pipeline profile list
    command.

  • --worker-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional
    non-heap memory per worker unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05
    (but this could be raised on a realm basis).

  • --garbage-collector-profile <profile ID> ID of the garbage collector profile that is used to configure the
    garbage collection strategy for the pipeline execution.
    To list all the currently available garbage collector profiles, enter the
    olp pipeline environment list command.

    Note::

    If garbage collector profile is not specified during the pipeline version creation, a profile
    from the appropriate pipeline template is used.

  • --billing-tag <tag,tag1,...> A comma separated tags for cost allocation (up to 6 billing tags), used to group billing records.

  • --runtime-config <key1=value1 key2=value2 ...> A custom map of key-value
    configuration for the pipeline application.
    Runtime configuration settings supplied here are added to any default runtime
    configuration settings from the parent pipeline template.
    Any runtime configuration settings here override any defaults from the parent
    pipeline template that are using the same key.

  • --stream-config <key1=value1 key2=value2 ...> A custom map of a key-value
    configuration for the stream pipeline version.
    With this parameter, you can pass key-value pairs to specify
    a custom configuration for stream-2.x.x and newer pipelines.
    For all supported parameters under this option, see
    Stream Configuration.

    Note

    The value of the --runtime-config and --stream-config parameters should be quoted with double quotes, like
    "key1=value1 key2=value2". In case some value contains spaces, it should be quoted with
    single quotes like "key1='some value1' key2=value2".
    Take into account that a parameter value with the \n splitter is also acceptable.
    Example: "key1='some value1'\nkey2=value2\n"

  • --multi-region For a pipeline that requires minimum downtime, use this flag
    to enable the multi-region setup for the pipeline version so that when
    the primary region fails, the pipeline version gets automatically transferred
    to the secondary region. To learn more about the multi-region functionality of pipelines,
    see Enable Multi-Region Setup for Pipelines

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.

  • --json Displays the pipeline version in JSON format.

  • --quiet Displays the pipeline version ID.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

Deprecated optional parameters:

  • --with-scheduler Sets up a scheduler to run the pipeline version when
    an input catalog changes. If not set, the pipeline version runs immediately
    after activation. This parameter is not applicable to a stream pipeline
    version. Deprecated.

    A new parameter, --schedule, was added to the
    olp pipeline version activate command to change the mode of execution of a
    pipeline version.

Note

HERE platform supports "Run As" to allow upgrading pipelines with a different
app. The OLP CLI always upgrades a pipeline using the app_id from the
credentials file. If you want to upgrade your pipeline with a different
app_id, use a different credentials file.

For more information on using credentials and profiles,
see Credentials setup.

For more information on pipeline configuration, see
Pipeline Configuration.

For more information on additional non-heap memory, see
Additional Non-Heap Memory.

Example:

The command below creates a pipeline version using the specified name,
pipeline ID, pipeline template ID, and configuration.

olp pipeline version create version-name f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   5c0660a3-0fb4-4f35-bcd0-be6ce25075f6 /user/data/pipeline-config.conf
olp pipeline version create version-name f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   5c0660a3-0fb4-4f35-bcd0-be6ce25075f6 /user/data/pipeline-config.conf

Output:

Pipeline version has been created
ID: d77f288e-2c89-4c94-b4ba-79fbd1e26e79

pipeline version activate

Activates the specified pipeline version and waits for the activate operation to
complete. If the operation is successful, the pipeline status becomes either
scheduled or running.

olp pipeline version activate <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --input-catalogs <input catalog specs...> Specific versions of input
    catalogs and execution mode for running a pipeline version job on demand
    only. This parameter applies to activating a batch pipeline version that needs
    to be executed with exact versions of input catalogs (defined in the
    pipeline-config.conf file
    and type of processing in the run-now mode without a scheduler (that is, with
    the --schedule parameter set to none). You are recommended to use
    --input-catalogs for testing and fine-tuning your pipeline version's run
    configuration locally on specific catalog versions before deploying it to the
    Here platform Pipelines API. For more information, see Data Processing Library examples
    that are part of the SDK for Java & Scala archive.

    This parameter accepts one of the following:

    • Path to the pipeline-job.conf file.
    • Input catalog versions and processing type in the
      catalog-id:processing-type:version[:since-version] format, where
      since-version is required only for the changes processing type.
      catalog-id is a fixed identifier used in both pipeline-config.conf and
      pipeline-job.conf. The processing-type describes what has changed in
      each input catalog since the last successful run; possible values are:
      no_changes, changes, reprocess.

    If --input-catalogs is not passed, the latest input catalog versions are
    automatically identified, and the reprocess processing type is applied.

  • --schedule <data-change|time:<cron-expression>|none> The execution mode
    of a batch pipeline version. The data-change value configures the pipeline
    version to run based on input catalog changes. The value none configures the
    pipeline version to run immediately. If this parameter is not used, the
    current mode of execution is used for activation. Check the current mode using
    the pipeline version show command. This parameter is not applicable to a
    stream pipeline version. The value time:<cron-expression> configures the
    pipeline version to run based on the schedule defined by a cron expression in
    the UNIX cron format.

  • --processing <changes|force> The processing type for the time-based schedule.
    _ Use changes (the default value) to first check for changes in the input catalog
    before running on the cron schedule. If there are changes in catalog versions,
    the system submits the job with the catalog versions.
    _ Use force to run the pipeline version on the given cron schedule without checking
    for changes in the input catalogs. If --input-catalogs is supplied,
    the system submits the job with provided catalog versions from the input with
    every run. If --input-catalogs is not supplied, the system submits the job
    without catalog versions.

    Note

    The processing parameter is only applicable for time-based scheduling, that is with --schedule time:cron-expression.

  • --high-availability The flag that enables a stream pipeline job to
    automatically recover from a failed Flink JobManager by using a secondary
    JobManager when the primary JobManager fails. This feature introduces additional
    cost for the extra resources. By default, the parameter value is false and
    a single JobManager is used for the stream pipeline job.

  • --snapshot-id The snapshot ID from a streaming pipeline. Resume a new job from the specified snapshot.
    For details, see the olp pipeline snapshot command.

    Note

    The --snapshot-id parameter is only applicable to streaming pipelines using runtime environment stream-5.0 or newer.

  • --with-latest-checkpoint The flag that starts a streaming pipeline from the latest checkpoint. If no checkpoint is found, the pipeline version activate operation fails.

    Note

    The --with-latest-checkpoint parameter is only applicable to streaming pipelines using runtime environment stream-5.0 or newer.

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.

  • --json Displays the activated pipeline version contents in JSON format.

  • --quiet Displays the pipeline version state.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

Warning

Backward compatibility

If a batch pipeline version is created using the deprecated
--with-scheduler parameter and then activated without the --schedule
parameter, it continues to work as expected. However, if the
--schedule none option is used during the activation, the pipeline version
gets reconfigured to run immediately (on demand). To change the mode back to
running based on input catalog changes, use --schedule data-change for the
next activation or use --schedule time:<cron-expression> to run
the pipeline version on a time-based schedule.

Note

The HERE platform supports "Run As" to allow upgrading pipelines with a different
app. The OLP CLI always upgrades a pipeline using the app_id from the
credentials file. If you want to upgrade your pipeline with a different
app_id, use a different credentials file.

For more information on using credentials and profiles,
see Credentials setup.

Example:

The command below activates the specified pipeline version.

olp pipeline version activate f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been activated
Current state: scheduled

If the pipeline version was created without the --with-scheduler parameter,
the pipeline version starts to run immediately, the pipeline log URL is
returned, and the link to the pipeline logs in Splunk is displayed.

If the pipeline version was created with the --with-scheduler parameter,
the pipeline version runs when the scheduler is triggered upon input catalog
changes.

Using pipeline-job.conf with --input-catalogs

An example of a pipeline-job.conf file is given below:

pipeline.job.catalog-versions {
    output-catalog { base-version = 1 }
    input-catalogs {
        catalogExample1 {
            processing-type = "no_changes"
            version = 1
        }
        catalogExample2 {
            processing-type = "no_changes"
            version = 1
        }
    }
}

For more information, see
Interfacing with the Pipeline Process.

pipeline version upgrade

Upgrades the specified pipeline version to a different pipeline version
and waits for the operation to complete.

olp pipeline version upgrade <pipeline ID> --to <pipeline version ID to upgrade to> \
   --from <pipeline version ID to upgrade from> [command option]
olp pipeline version upgrade <pipeline ID> --to <pipeline version ID to upgrade to> ^
   --from <pipeline version ID to upgrade from> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.

  • --to <pipeline version ID to upgrade to> The ID of the pipeline version to
    upgrade to.

  • --from <pipeline version ID to upgrade from> The ID of the pipeline version
    to upgrade from.

    Note

    The from pipeline version should be in the running or paused state when the command is executed.

    If the from batch pipeline version has been activated to run in On-demand (Run Now) mode,
    the upgrade operation will fail - only batch pipeline versions activated in Scheduled mode can be upgraded from.

Optional parameters:

  • --high-availability A flag that enables a stream pipeline job to
    automatically recover from a failed Flink JobManager by using a secondary
    JobManager when the primary JobManager fails. This feature introduces
    additional cost for the extra resources. By default, the parameter is false
    and a single JobManager is used for the stream pipeline job.

  • --schedule <data-change|time:<cron-expression>|none> The execution mode
    of a batch pipeline version. The data-change value configures the pipeline
    version to run based on input catalog changes. The time:<cron-expression> value configures the
    pipeline version to run based on the schedule defined by a cron expression in
    the UNIX cron format. The value none configures the
    pipeline version to run immediately. If this parameter is not used, the
    current mode of execution is used for activation. Check the current mode using
    the pipeline version show command. This parameter is not applicable to a
    stream pipeline version.

  • --processing <changes|force> The processing type for the time-based schedule.
    _ Use changes (the default value) to first check for changes in the input catalog
    before running on the cron schedule. If there are changes in catalog versions,
    the system submits the job with the catalog versions.
    _ Use force to run the pipeline version on the given cron schedule without checking
    for changes in the input catalogs. If --input-catalogs is supplied,
    the system submits the job with provided catalog versions from the input with
    every run. If --input-catalogs is not supplied, the system submits the job
    without catalog versions.

    Note

    The processing parameter is only applicable for time-based scheduling, that is with --schedule time:cron-expression.

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.

  • --json Displays the upgraded pipeline version contents in JSON format.

  • --quiet Displays the pipeline version state.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

Note

The HERE platform supports "Run As" to allow upgrading pipelines with a different
app. The OLP CLI always upgrades a pipeline using the app_id from the
credentials file. If you want to upgrade your pipeline with a different
app_id, use a different credentials file.

For more information on using credentials and profiles, see
Credentials setup.

Example:

The command below upgrades the pipeline version from the specified
version ID to a new version ID.

olp pipeline version upgrade f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 --to a21y693s-5f83-1h57-a0rr-28lrc5f10g93 \
   --from d77f288e-2c89-4c94-b4ba-79fbd1e26e79
olp pipeline version upgrade f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 --to a21y693s-5f83-1h57-a0rr-28lrc5f10g93 ^
   --from d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been upgraded
State: running

pipeline version pause

Pauses the specified pipeline version that was previously in the running
state. The pause operation follows different patterns for stream and batch
pipeline versions. For more information, see
Running a Pipeline.

olp pipeline version pause <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the paused pipeline version contents in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see
Credentials setup.

Example:

The command below pauses the specified pipeline version.

olp pipeline version pause f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been paused
State: paused

pipeline version resume

Resumes the specified paused pipeline version. A stream pipeline version
resumes from where it was paused, if possible. A batch pipeline version
resumes if and when a new job is available for it to process.

olp pipeline version resume <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --high-availability A flag that enables a stream pipeline job to
    automatically recover from a failed Flink JobManager by using a secondary
    JobManager when the primary JobManager fails. This feature introduces
    additional cost for the extra resources. By default, the parameter is false
    and a single JobManager is used for the stream pipeline job.
  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the resumed pipeline version contents in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see
Credentials setup.

Example:

The command below resumes the specified pipeline version.

olp pipeline version resume f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been resumed
State: scheduled

pipeline version list

Lists all pipeline versions belonging to a specified pipeline ID.

olp pipeline version list <pipeline ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the list of pipeline versions in JSON format.
  • --quiet Displays pipeline version IDs, each on a new line.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles,
see Credentials setup.

Example:

The command below lists all the versions belonging to the specified pipeline.

olp pipeline version list f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 --json

Output:

{"pipelineVersions": [{
    "multiRegionEnabled": false,
    "outputCatalogHRN": "hrn:here:data::org:test-catalog",
    "billingTag": null,
    "created": "2018-03-01T15:21:59.145Z",
    "versionNumber": 1,
    "clusterConfiguration": {
        "workerResourceProfileId": "HS1B",
        "supervisorResourceProfileId": "HS1B",
        "workerAdditionalOverheadMemory": 0,
        "gcProfile": "parallelgc",
        "supervisorUnits": 1,
        "workerUnits": 1,
        "workers": 1,
        "supervisorAdditionalOverheadMemory": 0
    },
    "name": "Version 1",
    "customRuntimeConfiguration": "",
    "schedulerConfiguration": {
        "catalogTriggerOptions": null,
        "timeTriggerOptions": {
            "cronJobExpression": "0 1 * * *",
            "processingType": "changes",
        }
    },
    "streamConfiguration": {"taskmanager.numberOfTaskSlots": 2},
    "id": "d77f288e-2c89-4c94-b4ba-79fbd1e26e79",
    "state": "ready",
    "pipelineTemplateId": "5c0660a3-0fb4-4f35-bcd0-be6ce25075f6",
    "loggingUrl": null,
    "updated": "2018-03-01T15:25:34.591Z",
    "inputCatalogs": {"input":
    "hrn:here:data::org:here-optimized-map-for-location-libraries-na-weu-1"}
}]}

pipeline version show

Shows all the information about the specified pipeline version.

olp pipeline version show <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the list of pipeline versions in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see
Credentials setup.

Example:

The command below displays information on the specified pipeline version.

olp pipeline version show f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json
olp pipeline version show f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json

Output:

{
    "multiRegionEnabled": false,
    "outputCatalogHRN": "hrn:here:data::org:test-catalog",
    "billingTag": null,
    "created": "2018-03-01T15:21:59.145Z",
    "versionNumber": 1,
    "clusterConfiguration": {
        "workerResourceProfileId": "HS1B",
        "supervisorResourceProfileId": "HS1B",
        "workerAdditionalOverheadMemory": 0,
        "gcProfile": "parallelgc",
        "supervisorUnits": 1,
        "workerUnits": 1,
        "workers": 1,
        "supervisorAdditionalOverheadMemory": 0
    },
    "name": "v1",
    "highAvailability": false,
    "customRuntimeConfiguration": "",
    "schedulerConfiguration": {
        "catalogTriggerOptions": null,
        "timeTriggerOptions": {
            "cronJobExpression": "0 1 * * *",
            "processingType": "changes",
        }
    },
    "nextJobAttempt": "2019-11-05T01:45:20Z"
    "streamConfiguration": {"taskmanager.numberOfTaskSlots": 2},
    "id": "d77f288e-2c89-4c94-b4ba-79fbd1e26e79",
    "state": "ready",
    "pipelineTemplateId": "5c0660a3-0fb4-4f35-bcd0-be6ce25075f6",
    "loggingUrl": null,
    "updated": "2018-03-01T15:25:34.591Z",
    "inputCatalogs": {"input":
    "hrn:here:data::org:here-optimized-map-for-location-libraries-na-weu-1"}
}

In the verbose output, the schedule field has replaced the
schedulerConfiguration field, and human-readable values such as none,
data-change, and <cron-expression> are now reported.
The next run attempt field shows the date and time when the next job will
be run for pipeline versions with the time-based schedule configuration.

Note

High availability

In the verbose output of the olp pipeline version show command, the
high availability status is shown as true if the pipeline version has
been activated, resumed, or upgraded with the --high-availability flag
and has a job in states running or starting.

pipeline version delete

Deletes the specified pipeline version if that pipeline version
is in the ready state. This means that to delete a running or paused
pipeline, you need to cancel it first.

olp pipeline version delete <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --force Cancels and deactivates a pipeline version, if needed,
    before deletion.
  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --quiet Displays empty output with no additional information.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles,
see Credentials setup.

Example:

The command below deletes the specified pipeline version.

olp pipeline version delete f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version d77f288e-2c89-4c94-b4ba-79fbd1e26e79 has been deleted

pipeline version cancel

Cancels the specified running pipeline version and any future jobs scheduled for
that pipeline version.

olp pipeline version cancel <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the pipeline version contents after
    cancellation in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see
Credentials setup.

Example:

The command below cancels the specified pipeline version.

olp pipeline version cancel f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been canceled
State: ready

pipeline version deactivate

Deactivates the specified activated pipeline version and waits
for the operation to complete.

olp pipeline version deactivate <pipeline ID> <pipeline version ID> [command option]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the deactivated pipeline version contents in JSON format.
  • --quiet Displays the pipeline version state.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see
Credentials setup.

Example:

The command below deactivates the specified pipeline version.

olp pipeline version deactivate f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79

Output:

Pipeline version has been deactivated
State: ready

pipeline version wait

Waits for the pipeline version to reach the specified job state.
If the pipeline version is in the ready or paused state, the command does
not wait for the timeout. Instead, it checks that the job state is reached.

olp pipeline version wait <pipeline ID> <pipeline version ID> \
   --job-state=completed [command options]
olp pipeline version wait <pipeline ID> <pipeline version ID> ^
   --job-state=completed [command options]

Once the job state is reached, the command displays the
pipeline job information. Otherwise, an error message is displayed.

Optional parameters:

  • --job-state <running|completed|failed|canceled> The job state for the
    pipeline version. If not specified, the default is completed.
  • --timeout <duration in seconds> The timeout duration in seconds. If not
    specified, the default is 3600.
  • --no-error Set this flag to override the state not reached timeout error.
  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the pipeline version job in JSON format if the command
    succeeds.
  • --quiet Displays empty output with no additional information.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information, see
Pipeline States.

For more information on using credentials and profiles,
see Credentials setup.

Example:

The command below waits for a pipeline job to complete.

olp pipeline version wait f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 \
   --job-state=completed --json
olp pipeline version wait f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 ^
   --job-state=completed --json

Output:

{
  "catalogVersions": null,
  "created": "2018-03-01T15:23:04.618Z",
  "id": "e8e31070-4f4b-4e4b-a8e0-40425844cb75",
  "state": "failed",
  "updated": "2018-03-01T15:23:32.497Z",
  "loggingUrl": null
}

pipeline version copy

Copies a specified pipeline version and its logging configuration. The copied
version is in the ready state and can be later activated manually. Use this
command to speed up the process of configuring your pipelines.

olp pipeline version copy <pipeline ID> <pipeline version ID> [command options]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --jar <file path> A fat JAR file to upload, including its path on a local
    file system. If provided, the pipeline version is created with this template.
    For more information on the JAR file contents and build instructions, have a look at our tutorial on
    Running a Stream Application with Pipeline Service.

  • --pipeline-config <file path> A configuration file with the specified
    output and input catalogs.

  • --with-scheduler Sets up a scheduler to run the pipeline version when
    an input catalog changes. If not set, the pipeline version runs immediately
    after activation. This parameter does not apply for a stream environment.
    Deprecated.

    A new parameter, --schedule, was added to the
    olp pipeline version activate command to change the mode of execution of the
    pipeline version.

  • --supervisor-units <size in units> The size of a supervisor node
    (1 to 15 units). If not specified, the value is taken from the original pipeline version.

  • --supervisor-units-profile <resource profile ID> The ID of the resource profile requested for the supervisor units.
    If not specified, the value is taken from the original pipeline version. To list all the currently available
    resource profiles, enter the olp pipeline profile list
    command.

  • --supervisor-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional
    non-heap memory per supervisor unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05
    (but this could be raised on a realm basis).

  • --workers <number of workers> The number of workers. If not specified, the value is taken from the original
    pipeline version.

  • --worker-units <size in units> The size of a worker node (1 to 15 units).
    If not specified, the value is taken from the original pipeline version.

  • --worker-units-profile <resource profile ID> ID of the resource profile requested for the worker units.
    If not specified, the value is taken from the original pipeline version. To list all the currently available
    resource profiles, enter the olp pipeline profile list
    command.

  • --worker-additional-overhead-memory <overhead memory in fraction> The factor of proportionality of additional
    non-heap memory per worker unit. The value can be greater than or equal to 0.00 with default maximum value of 0.05
    (but this could be raised on a realm basis).

  • --garbage-collector-profile <profile ID> ID of the garbage collector profile that is used to configure the
    garbage collection strategy for the pipeline execution.
    To list all the currently available garbage collector profiles, enter the
    olp pipeline environment list command.

    Note::

    If garbage collector profile is not specified during the pipeline version creation, a profile
    from the appropriate pipeline template is used.

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.

  • --json Displays the pipeline version job in JSON format if the command runs
    successfully.

  • --quiet Displays the pipeline version ID.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles,
see Credentials setup.

For more information on pipeline configuration, see
Interfacing with the Pipeline Process.

For more information on additional non-heap memory, see
Additional Non-Heap Memory.

Example:

The command below copies a pipeline version and customizes the copied
pipeline version with a new configuration file and fat JAR file.

olp pipeline version copy f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 \
   --pipeline-config example-pipeline-configuration.conf --jar example.jar
olp pipeline version copy f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 ^
   --pipeline-config example-pipeline-configuration.conf --jar example.jar

Output:

Pipeline version has been copied
ID: 143ww7ux-101d-46c4-8bdd-f8919067bba4

pipeline version log level get

Shows the current logging configuration for a specified pipeline version.

olp pipeline version log level get <pipeline ID> <pipeline version ID> [command parameters]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the logging configuration in JSON format.
  • --quiet Displays the logging levels for the specified pipeline version
    separated by space.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using logs, see
Logs, Monitoring, and Alerts User Guide.

For more information on using credentials and profiles, see
Credentials setup.

Example:

olp pipeline version log level get \
   f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json
olp pipeline version log level get ^
   f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json

Output:

{"configuration": {"loggers": {
    "logger": [
        {
            "level": "info",
            "name": "com.example.someLoggingClass"
        },
        {
            "level": "info",
            "name": "com.example.examplepkg.otherClass"
        }
    ],
    "root": {"level": "error"}
}}}

pipeline version log level set

Sets the current logging configuration for a specified pipeline version.

olp pipeline version log level set <pipeline-id> <pipeline-version-id> [command parameters]

Required parameters:

  • <pipeline ID> The ID of the pipeline.
  • <pipeline version ID> The ID of the pipeline version.
  • --log4j-properties <property file> Set the logging levels in the
    log4j format from the
    property file (see below for a sample property file content).
  • --root <log level> Set the root level to one of the four values: debug,
    info, warn, or error. The root logger can be configured through the
    command line. However, the use of this parameter doesn't overwrite the existing
    configuration set for each logger.

Warning

Setting the log level

To set the log level, use either the --log4j-properties or the root
parameter.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use
    with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the
    olpcli.ini file.
  • --json Displays the logging configuration in JSON format.
  • --quiet Displays the logging levels for the specified pipeline versions,
    separated by space.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the
    request. The value specified with --scope overrides any value for
    here.token.scope provided in the credentials file used for the command.

For more information on using logs, see
Logs, Monitoring and Alerts User Guide.

For more information on using credentials and profiles, see
Credentials setup.

Example:

olp pipeline version log level set f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 \
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --root DEBUG --json
olp pipeline version log level set f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 ^
   d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --root DEBUG --json

Output:

{
  "configuration": {
    "loggers": {
      "root": { "level": "debug" }
    }
  }
}

You can configure the root and separate loggers by providing log4j 1.x
properties in the property file; only rootLogger and logger
properties are consumed.

Example of a log4j property file:

# initialize root logger with level ERROR
log4j.rootLogger=ERROR
# set the log level for these components
log4j.logger.com.example.someLoggingClass=INFO
log4j.logger.com.example.examplepkg.otherClass=INFO

For more information on log4j, see the
log4j documentation.