GuidesChangelogData Inspector Library API Reference
Guides
  1. Command Line Interface
  2. OLP CLI Command reference

catalog layer partition

catalog layer partition

The OLP CLI supports the following:

  • list partitions in a layer
  • get partitions from a layer
  • put partitions in a layer
  • delete partitions in a layer

catalog layer partition list

Lists the names of all partitions in the specified catalog layer; either versioned, volatile, or index.

olp catalog layer partition list <catalog HRN> <layer ID> [command options]

Note

The olp catalog layer partition list command output is limited to 100,000 partitions. If this limit is exceeded, the following warning message appears: "WARNING: The limit of 100,000 partitions has been reached" The warning message appears only if the command has been executed without the --json or --quiet flag.

Required parameters:

  • <catalog HRN> The HRN of the catalog.

  • <layer ID> The ID of the layer; either versioned, volatile, or index. For stream layers, see Stream commands

  • --filter <filter query> The query string required for Index layers. The <filter query> must be written in Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside the <filter query> should be escaped or quoted.

    Note

    --filter <filter query> is optional for versioned and volatile layers, where it is used as a prefix to filter partitions by partition name.

Optional parameters:

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --json Displays the command result in JSON format.
  • --quiet Displays partition names, each on a new line.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

Optional parameters for a versioned layer:

  • --version <catalog version> The catalog's metadata version.
  • --change startVersion:[endVersion] Get metadata for the latest partition changes for a version range in a versioned layer. Only the latest change for each changed partition is returned, not intermediate changes. If the response contains a partition with an empty dataHandle value, it means that the partition was deleted. If a partition was created and deleted between the versions specified in the startVersion and endVersion parameters, the partition is included in the response with an empty dataHandle value. For example, if you request versions 0 to 10, and a partition was created in version 3 and then deleted in version 7, it is included in the response with an empty dataHandle value. If endVersion is not specified, the latest version of the catalog metadata will be used.
  • --context <default|super|extension> The context where the operation will be performed on composite layers. Available context are:
    • default For composite layers, the operation is executed for both the current layer and the layer it extends. For normal layers, this is the only valid context.
    • extension The operation will be executed only in the extension, and no operation will be performed in the extended layer.
    • super The operation will be executed only in the layer being extended.

Optional parameters for a volatile layer:

  • --modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile layers based on when they were last modified. The timestamp must match the ISO-8601 format YYYY-MM-DDThh:mm:ss but only YYYY is mandatory. To skip values in the timestamp, you must start from the least significant value, from right to left.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below lists all partitions from layer1 in the catalog catalog1.

olp catalog layer partition list hrn:here:data::org:catalog1 layer1

Output:

partition                    size                dataHandle                              checksum                                          CRC
testPartition                23                  3f24f359-4b6d-4d51-a39f-1140r112w111

Total size: 23 B
Use olp catalog layer partition get <catalog HRN> <layer ID> to download partitions

catalog layer partition get

Downloads the partitions that are specified with the --partitions parameter in the catalog layer.

olp catalog layer partition get <catalog HRN> <layer ID> [command options]

Note

Partition(s) will be automatically decompressed during olp catalog layer partition get command execution if the layer was created with the --content-encoding <gzip> parameter.

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to use; either versioned, volatile, or index. For stream layers, see catalog layer stream commands.

Required parameters for an index layer:

  • --filter <filter query> The query string required for Index layers. The <filter query> must be written in Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside the <filter query> should be escaped or quoted. This parameter is optional for versioned and volatile layers, where it is used as a prefix to filter the partitions by partition name.

Optional parameters:

  • --partitions <partitionID1 partitionID2 ...> The partitions to download. If only one partition is specified and no output folder is provided, the partition content is retrieved into the standard output. If the partition content is retrieved into the standard output, the --modified-since parameter is used, and there are no modifications after the date provided, then the command execution finishes with the exit code 4.

    Note

    The command can produce empty output when trying to write the content of the partition that contains non-ASCII symbols to stdout. This is a Windows-specific problem, related to the peculiarity of rendering UTF-8 symbols on Windows console. As a workaround, please redirect command output to a file or specify the output directory with the --output parameter.

  • --all Downloads all partitions in the layer.

  • --parallelism <number> Configure the number of concurrent download requests.

    Note

    When processing large numbers of partitions, command now automatically switches to PERFORMANCE client configuration mode, which optimizes HTTP connection pooling and allows more open connections.

  • --output <path to dir> The output directory to store the downloaded partitions. If the specified directory doesn't exist, it is automatically created. If no directory is specified, the default value is <current dir>/<catalog HRN>.

  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.

  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.

  • --quiet Displays the absolute path to the output directory.

  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

Note

For versioned or volatile layers, you must specify either the --all or the --partitions parameter.

Optional parameters for a versioned layer:

  • --version <catalog version> The catalog's metadata version.
  • --decode <true|false> Determines whether the partition content is decoded from protobuf and converted to JSON on-the-fly. The default value is false, which means the content is not decoded. You can specify the path to a schema on a local file system with --decode <true|false:[path/to/schema.zip]>; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command. Using --decode true with --output or --all parameters produces files where the following name pattern is used: <partition-name>-decoded.json.
  • --context <default|super|extension> The context where the operation will be performed on composite layers. Available context are:
    • default For composite layers, the operation is executed for both the current layer and the layer it extends. For normal layers, this is the only valid context.
    • extension The operation will be executed only in the extension, and no operation will be performed in the extended layer.
    • super The operation will be executed only in the layer being extended.

Optional parameters for a volatile layer:

  • --modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile layers based on when they were last modified. The timestamp must match the ISO-8601 format YYYY-MM-DDThh:mm:ss, but only YYYY is mandatory. To skip values in the timestamp, you must start from the least significant value, from right to left.
  • --decode <true|false> Determines whether the partition content is decoded from protobuf and converted to JSON on-the-fly. The default value is false, which means the content is not decoded. You can specify the path to a schema on a local file system with --decode <true|false:[path/to/schema.zip]>; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command. Using --decode true with --output or --all parameters produces files where the following name pattern is used: <partition-name>-decoded.json.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below retrieves the partition content into the standard output if you specify only one partition.

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 \
   --partitions partition1
olp catalog layer partition get hrn:here:data::org:catalog1 layer1 ^
   --partitions partition1

Output:

Partition content example

Example:

The command below downloads partition1 from layer1 in catalog1 to the specified output file.

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 \
   --partitions partition1 > PATH/TO/OUTPUT_FILE
olp catalog layer partition get hrn:here:data::org:catalog1 layer1 ^
   --partitions partition1 > PATH/TO/OUTPUT_FILE

Example:

The command below downloads partition1 and partition2 from version 1 of versioned-layer in catalog1 to the specified output directory.

olp catalog layer partition get hrn:here:data::org:catalog1 versioned-layer \
   --partitions partition1 partition2 --output PATH/TO/OUTPUT_DIRECTORY --version 1
olp catalog layer partition get hrn:here:data::org:catalog1 versioned-layer ^
   --partitions partition1 partition2 --output PATH/TO/OUTPUT_DIRECTORY --version 1

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions: partition1,partition2 were successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Example:

To customize the path where the partition is saved, concatenate the relative or the absolute path with the partition name, using a colon (:).

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 \
   --partitions partition_name:PATH/TO/OUTPUT_DIRECTORY
olp catalog layer partition get hrn:here:data::org:catalog1 layer1 ^
   --partitions partition_name:PATH/TO/OUTPUT_DIRECTORY

Output:

0% [>                                                  ] 0/1 (0:00:00 / ?)
0% [>                                                  ] 0/1 (0:00:00 / ?)
100% [=============================================] 1/1 (0:00:00 / 0:00:00)

Partition partition_name was successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Example:

The command below retrieves all partitions from the index layer that were uploaded on January 1, 2020, between 00:01:00 AM and 00:05:00 AM with locationTag equal to 4242.

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 \
   --filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
   --output PATH/TO/OUTPUT_DIRECTORY
olp catalog layer partition get hrn:here:data::org:catalog1 layer1 ^
   --filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
   --output PATH/TO/OUTPUT_DIRECTORY

Output:

0% [>                                                  ] 0/3 (0:00:00 / ?)
0% [>                                                  ] 0/3 (0:00:00 / ?)
0% [>                                                  ] 0/3 (0:00:00 / ?)
100% [=============================================] 3/3 (0:00:00 / 0:00:00)

Partitions: partition1,partition2,partition3 were successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Note

The --filter value depends on the --index-fields value that was specified during partitions upload (olp catalog layer partition put command execution). Also, note that the --index-fields value depends on the --index-definitions value specified during index layer creation - catalog layer add command execution.

Query Performance Tips

  1. Use the timewindow indexing attribute in the query.
  2. To get results back within a short period of time, you should split queries based on timewindow so that the data in the timewindow range is manageable (100,000 or less).
  3. Limit the usage of additional constraints.

catalog layer partition put

Publishes the specified files to the specified partitions.

olp catalog layer partition put <catalog HRN> <layer ID> [command options]

Note

Partition(s) content will be automatically compressed during the olp catalog layer partition put command execution if the layer was created with the --content-encoding <gzip> parameter.

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to upload partition(s) to; either versioned, volatile, or index. For stream layers, see catalog layer stream commands.

Required parameters for an index layer:

  • --index-fields <bool|int|string|heretile|timewindow>:<index-key>:<index-value> A list of index fields for this publication. The <timewindow> index field definition is required and should be the same as was specified in the layer's index definitions.

    If the --index-fields parameter is not passed and the --input parameter is passed, the application tries to read the index configuration from the index.properties file located in the --input path. The format of the index.properties file is the following:

    • fileMask A regular expression used to filter files in the input directory.
    • indexType An index type name with these acceptable values: bool, int, string, heretile, timewindow. You must define timewindow that should be the same as the one specified in the layer index-definitions.
    • indexKey An index name. This key is used for listing index partitions.
    • indexValue The value of the index. The value of timewindow should be the same as the one specified in the layer index-definitions.

    The following example sets the timewindow index myIndex to 50000 for all the files whose names start with file and one digit like file1.bin, file2.txt, and the like. Add the boolean index key myBoolIndex set to true to the file1.bin file.

    fileMask=file[0-9].*
    indexType=timewindow
    indexKey=myIndex
    indexValue=50000

    fileMask=file1.bin
    indexType=bool
    indexKey=myBoolIndex
    indexValue=true

An empty line between index definitions is not required but it improves the readability of the config. The values for all configuration keys (fileMask, indexType, indexKey, indexValue) are required.

If both of the --index-fields and --input parameters are passed, then the configurations from index.properties are ignored.

Optional parameters:

  • --partitions <partition ID>:<path to file> The partitions and files to upload.

  • --parallelism <number> Configure the number of concurrent upload requests.

    Note

    When processing large numbers of partitions, command now automatically switches to PERFORMANCE client configuration mode, which optimizes HTTP connection pooling and allows more open connections.

  • --input <path to dir>/<glob-pattern> The input directory, single file, or filename glob pattern (non-recursive) from which to upload the files; supports relative paths; hidden files are skipped.

Note

If neither --input nor --partitions parameters are specified, nothing will be uploaded, but metadataVersion will be incremented.

If you wish to use both the --partitions and --input options, specify the directory containing the files you wish to put with the --input option. Then, specify the relative paths to the individual files you wish to put from that directory using the --partitions parameter.

  • --dependency <catalog-hrn:version[:direct]> List of the dependencies for this publication where direct is either true or false, and its default value is true.
  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --quiet Displays empty output with no additional information.
  • --scope <project HRN> Specifies the project HRN to use as the scope in the request. The value specified with --scope overrides any value for here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see Credentials setup.

Optional parameters for a versioned layer:

  • --encode <true|false:[path/to/schema.zip]> Determines whether the partition content is encoded from JSON and converted to protobuf on-the-fly. The default value is false, which means the content is not encoded. You can specify the path to a schema on a local file system; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command.

Note

You can encode only the JSON file that you got from the olp catalog layer partition get command.

Optional parameters for a volatile layer:

  • --encode <true|false:[path/to/schema.zip]> Determines whether the partition content is encoded from JSON and converted to protobuf on-the-fly. The default value is false, which means the content is not encoded. You can specify the path to a schema on a local file system; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command.

Note

You can encode only the JSON file that you got from the olp catalog layer partition get command.

Optional parameters for an index layer:

  • --metadata <key1=val1 key2=val2 ...> List of all user-defined fields that can store extra information. If the --input parameter is provided, the metadata applies to all the files located in the input directory.

Example:

The command below uploads file1 and file2 to partition1 and partition2 respectively from the folder relative to the current folder.

olp catalog layer partition put hrn:here:data::org:catalog1 layer1 \
   --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
olp catalog layer partition put hrn:here:data::org:catalog1 layer1 ^
   --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions partition1, partition2 were successfully uploaded

Example:

If you specify only one partition, the command retrieves the partition content into the standard output. For more information, see olp catalog layer partition get.

OLP CLI lets you chain both get and put commands when you retrieve content into the standard output. This means that you can omit the path to the partition and use a hyphen (-) instead.

olp catalog layer partition get hrn:here:data::org:catalog-1 versioned-layer1 \
   --partitions partition1 | olp catalog layer partition put hrn:here:data::org:catalog-2 \
   versioned-layer1 --partitions partition1:-
olp catalog layer partition get hrn:here:data::org:catalog-1 versioned-layer1 ^
   --partitions partition1 | olp catalog layer partition put hrn:here:data::org:catalog-2 ^
   versioned-layer1 --partitions partition1:-

Output:

0% [>                                                  ] 0/1 (0:00:00 / ?)
0% [>                                                  ] 0/1 (0:00:00 / ?)
100% [=============================================] 1/1 (0:00:00 / 0:00:00)

Partition partition1 was successfully uploaded

Example:

The command below uploads file1 and file2 to partition1 and partition2 respectively to the index layer with the --index-fields parameter equal to timewindow:ingestionTime:1577836960 int:locationTag:4242

Note

The --index-fields value depends on the --index-definitions value specified during index layer creation - olp catalog layer add command execution.

olp catalog layer partition put hrn:here:data::org:catalog1 layer1 \
   --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
   --index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242
olp catalog layer partition put hrn:here:data::org:catalog1 layer1 ^
   --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
   --index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions partition1, partition2 were successfully uploaded

catalog layer partition delete

Deletes partitions from an index layer by the --filter query. Note that this command does not support data deletion from versioned and volatile layers.

olp catalog layer partition delete <catalog HRN> <layer ID> [command options]

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to delete partitions from.
  • --filter <filter query> The query string required to query data that should be deleted from the index layer. The <filter query> value must be written in the Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside <filter query> should be escaped or quoted.

Optional parameters:

  • --timeout <duration in seconds> The timeout duration of the delete request in seconds. If not specified, the default is 3600.
  • --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
  • --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
  • --quiet Displays empty output with no additional information.

Example:

The command below deletes partitions with keys testIndex==3600000 and env==4242.

olp catalog layer partition delete hrn:here:data::org:catalog1 indexLayer1 \
   --filter "testIndex==3600000;env==4242"
olp catalog layer partition delete hrn:here:data::org:catalog1 indexLayer1 ^
   --filter "testIndex==3600000;env==4242"

Output:

Please wait while the partitions are being deleted. This may take several minutes
1 partition has been successfully deleted

For more information on using credentials and profiles, see Credentials setup.

Updated 21 days ago