GuidesChangelogData Inspector Library API Reference
Guides
  1. Command Line Interface
  2. OLP CLI Command reference

local catalog layer partition

local catalog layer partition

The OLP CLI supports the following:

  • list partitions in a local layer
  • get partitions from a local layer
  • put partitions in a local layer
  • delete partitions in a local layer

local catalog layer partition list

Lists the names of all partitions in the specified catalog layer; either versioned, volatile, or index.

olp local catalog layer partition list <catalog HRN> <layer ID> [command options]

Note

The olp catalog layer partition list command output is limited to 100,000 partitions. If this limit is exceeded, the following warning message appears: "WARNING: The limit of 100,000 partitions has been reached" The warning message appears only if the command has been executed without the --json or --quiet flag.

Required parameters:

  • <catalog HRN> The HRN of the catalog.

  • <layer ID> The ID of the layer; either versioned, volatile, or index. For stream layers, see local catalog layer stream commands.

  • --filter <filter query> The query string required for Index layers. The <filter query> must be written in Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside the <filter query> should be escaped or quoted.

    Note

    --filter <filter query> is optional for versioned and volatile layers, where it is used as a prefix to filter partitions by partition name.

Optional parameters:

  • --json Displays the command result in JSON format.
  • --quiet Displays partition names, each on a new line.

Optional parameters for a versioned layer:

  • --version <catalog version> The catalog's metadata version.

Optional parameters for a volatile layer:

  • --modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile layers based on when they were last modified. The timestamp must match the ISO-8601 format YYYY-MM-DDThh:mm:ss but only YYYY is mandatory. To skip values in the timestamp, you must start from the least significant value, from right to left.

Example:

The command below lists all partitions from layer1 in the catalog catalog1.

olp local catalog layer partition list hrn:local:data:::catalog1 layer1

Output:

partition                    size                dataHandle                              checksum                                          CRC
testPartition                23                  415efy1d-5d50-27ge-5876-84a26f2a6a43

Total size: 23 B
Use olp local catalog layer partition get <catalog HRN> <layer ID> to download partitions

local catalog layer partition get

Downloads the partitions that are specified with the --partitions parameter in the local catalog layer.

olp local catalog layer partition get <catalog HRN> <layer ID> [command options]

Note

Partition(s) will be automatically decompressed during olp local catalog layer partition get command execution if the layer was created with the --content-encoding <gzip> parameter.

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to use; either versioned, volatile, or index. For stream layers, see local catalog layer stream commands.

Required parameters for an index layer:

  • --filter <filter query> The query string required for Index layers. The <filter query> must be written in Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside the <filter query> should be escaped or quoted. This parameter is optional for versioned and volatile layers, where it is used as a prefix to filter the partitions by partition name.

Optional parameters:

  • --partitions <partitionID1 partitionID2 ...> The partitions to download. If only one partition is specified and no output folder is provided, the partition content is retrieved into the standard output. If the partition content is retrieved into the standard output, the --modified-since parameter is used, and there are no modifications after the date provided, then the command execution finishes with the exit code 4.
  • --all Downloads all partitions in the layer.
  • --output <path to dir> The output directory to store the downloaded partitions. If the specified directory doesn't exist, it is automatically created. If no directory is specified, the default value is <current dir>/<catalog HRN>.
  • --quiet Displays the absolute path to the output directory.

Note

For versioned or volatile layers, you must specify either the --all or the --partitions parameter.

Optional parameters for a versioned layer:

  • --version <catalog version> The catalog's metadata version.
  • --decode <true|false> Determines whether the partition content is decoded from protobuf and converted to JSON on-the-fly. The default value is false, which means the content is not decoded. You can specify the path to a schema on a local file system with --decode <true|false:[path/to/schema.zip]>; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command. Using --decode true with --output or --all parameters produces files where the following name pattern is used: <partition-name>-decoded.json.

Optional parameters for a volatile layer:

  • --modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile layers based on when they were last modified. The timestamp must match the ISO-8601 format YYYY-MM-DDThh:mm:ss, but only YYYY is mandatory. To skip values in the timestamp, you must start from the least significant value, from right to left.
  • --decode <true|false> Determines whether the partition content is decoded from protobuf and converted to JSON on-the-fly. The default value is false, which means the content is not decoded. You can specify the path to a schema on a local file system with --decode <true|false:[path/to/schema.zip]>; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command. Using --decode true with --output or --all parameters produces files where the following name pattern is used: <partition-name>-decoded.json.

Example:

The command below retrieves the partition content into the standard output if you specify only one partition.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition1
olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition1

Output:

Local partition content example

Example:

The command below downloads partition1 from layer1 in catalog1 to the specified output file.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition1 > PATH/TO/OUTPUT_FILE
olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition1 > PATH/TO/OUTPUT_FILE

Example:

The command below downloads partition1 and partition2 from version 1 of versioned-layer in catalog1 to the specified output directory.

olp local catalog layer partition get hrn:local:data:::catalog1 versioned-layer \
 --partitions partition1 partition2 --output PATH/TO/OUTPUT_DIRECTORY --version 1
olp local catalog layer partition get hrn:local:data:::catalog1 versioned-layer ^
--partitions partition1 partition2 --output PATH/TO/OUTPUT_DIRECTORY --version 1

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions: partition1,partition2 were successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Example:

To customize the path where the partition is saved, concatenate the relative or the absolute path with the partition name, using a colon (:).

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition_name:PATH/TO/OUTPUT_DIRECTORY
olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition_name:PATH/TO/OUTPUT_DIRECTORY

Output:

0% [>                                                  ] 0/1 (0:00:00 / ?)
0% [>                                                  ] 0/1 (0:00:00 / ?)
100% [=============================================] 1/1 (0:00:00 / 0:00:00)

Partition partition_name was successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Example:

The command below retrieves all partitions from the index layer that were uploaded on January 1, 2020, between 00:01:00 AM and 00:05:00 AM with locationTag equal to 4242.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
--output PATH/TO/OUTPUT_DIRECTORY
olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
--output PATH/TO/OUTPUT_DIRECTORY

Output:

0% [>                                                  ] 0/3 (0:00:00 / ?)
0% [>                                                  ] 0/3 (0:00:00 / ?)
0% [>                                                  ] 0/3 (0:00:00 / ?)
100% [=============================================] 3/3 (0:00:00 / 0:00:00)

Partitions: partition1,partition2,partition3 were successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Note

The --filter value depends on the --index-fields value that was specified during partitions upload (olp local catalog layer partition put command execution). Also, note that the --index-fields value depends on the --index-definitions value specified during index layer creation - catalog layer add command execution.

local catalog layer partition put

Publishes the specified files to the specified local partitions.

olp local catalog layer partition put <catalog HRN> <layer ID> [command options]

Note

Partition(s) content will be automatically compressed during the olp local catalog layer partition put command execution if the layer was created with the --content-encoding <gzip> parameter.

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to upload partition(s) to; either versioned, volatile, or index. For stream layers, see local catalog layer stream commands.

Required parameters for an index layer:

  • --index-fields <bool|int|string|heretile|timewindow>:<index-key>:<index-value> A list of index fields for this publication. The <timewindow> index field definition is required and should be the same as was specified in the layer's index definitions.

    If the --index-fields parameter is not passed and the --input parameter is passed, the application tries to read the index configuration from the index.properties file located in the --input path. The format of the index.properties file is the following:

    • fileMask A regular expression used to filter files in the input directory.
    • indexType An index type name with these acceptable values: bool, int, string, heretile, timewindow. You must define timewindow that should be the same as the one specified in the layer index-definitions.
    • indexKey An index name. This key is used for listing index partitions.
    • indexValue The value of the index. The value of timewindow should be the same as the one specified in the layer index-definitions.

    The following example sets the timewindow index myIndex to 50000 for all the files whose names start with file and one digit like file1.bin, file2.txt, and the like. Add the boolean index key myBoolIndex set to true to the file1.bin file.

    fileMask=file[0-9].*
    indexType=timewindow
    indexKey=myIndex
    indexValue=50000

    fileMask=file1.bin
    indexType=bool
    indexKey=myBoolIndex
    indexValue=true

An empty line between index definitions is not required but it improves the readability of the config. The values for all configuration keys (fileMask, indexType, indexKey, indexValue) are required.

If both of the --index-fields and --input parameters are passed, then the configurations from index.properties are ignored.

Optional parameters:

  • --partitions <partition ID>:<path to file> The partitions and files to upload.
  • --input <path to dir> The input directory from which to upload the files; supports relative paths; hidden files are skipped.
  • --dependency <catalog-hrn:version[:direct]> The list of the dependencies for this publication. direct is optional, and its default value is true.
  • --quiet Displays empty output with no additional information.

Note

If you wish to use both the --partitions and --input options, specify the directory containing the files you wish to put with the --input option. Then, specify the relative paths to the individual files you wish to put from that directory using the --partitions parameter.

Optional parameters for a versioned layer:

  • --encode <true|false:[path/to/schema.zip]> Determines whether the partition content is encoded from JSON and converted to protobuf on-the-fly. The default value is false, which means the content is not encoded. You can specify the path to a schema on a local file system; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command.

Note

You can encode only the JSON file that you got from the olp local catalog layer partition get command.

Optional parameters for a volatile layer:

  • --encode <true|false:[path/to/schema.zip]> Determines whether the partition content is encoded from JSON and converted to protobuf on-the-fly. The default value is false, which means the content is not encoded. You can specify the path to a schema on a local file system; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command.

Note

You can encode only the JSON file that you got from the olp local catalog layer partition get command.

Optional parameters for an index layer:

  • --metadata <key1=val1 key2=val2 ...> List of all user-defined fields that can store extra information. If the --input parameter is provided, the metadata applies to all the files located in the input directory.

Example:

The command below uploads file1 and file2 to partition1 and partition2 respectively from the folder relative to the current folder.

olp local catalog layer partition put hrn:local:data:::catalog1 layer1 \
 --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
olp local catalog layer partition put hrn:local:data:::catalog1 layer1 ^
--partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions partition1, partition2 were successfully uploaded

Example:

If you specify only one partition, the command retrieves the partition content into the standard output. For more information, see olp local catalog layer partition get.

OLP CLI lets you chain both get and put commands when you retrieve content into the standard output. This means that you can omit the path to the partition and use a hyphen (-) instead.

olp local catalog layer partition get hrn:local:data:::catalog-1 versioned-layer1 \
 --partitions partition1 | olp local catalog layer partition put hrn:local:data:::catalog-2 \
 versioned-layer1 --partitions partition1:-
olp local catalog layer partition get hrn:local:data:::catalog-1 versioned-layer1 ^
--partitions partition1 | olp local catalog layer partition put hrn:local:data:::catalog-2 ^
versioned-layer1 --partitions partition1:-

Output:

0% [>                                                  ] 0/1 (0:00:00 / ?)
0% [>                                                  ] 0/1 (0:00:00 / ?)
100% [=============================================] 1/1 (0:00:00 / 0:00:00)

Partition partition1 was successfully uploaded

Example:

The command below uploads file1 and file2 to partition1 and partition2 respectively to the index layer with the --index-fields parameter equal to timewindow:ingestionTime:1577836960 int:locationTag:4242

Note

The --index-fields value depends on the --index-definitions value specified during index layer creation - catalog layer add command execution.

olp local catalog layer partition put hrn:local:data:::catalog1 layer1 \
 --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
--index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242
olp local catalog layer partition put hrn:local:data:::catalog1 layer1 ^
--partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
--index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions partition1, partition2 were successfully uploaded

local catalog layer partition delete

Deletes partitions from an index layer by the --filter query. Note that this command does not support data deletion from versioned and volatile layers.

olp local catalog layer partition delete <catalog HRN> <layer ID> [command options]

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to delete partitions from.
  • --filter <filter query> The query string required to query data that should be deleted from the index layer. The <filter query> value must be written in the Feed Item Query Language. We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators instead of the <, <=, >, or >= operators on Windows. All special characters inside <filter query> should be escaped or quoted.

Optional parameters:

  • --timeout <duration in seconds> The timeout duration of the delete request in seconds. If not specified, the default is 3600.
  • --quiet Displays empty output with no additional information.

Example:

The command below deletes partitions with keys testIndex==3600000 and env==4242.

olp local catalog layer partition delete hrn:local:data:::catalog1 indexLayer1 \
 --filter "testIndex==3600000;env==4242"
olp local catalog layer partition delete hrn:local:data:::catalog1 indexLayer1 ^
--filter "testIndex==3600000;env==4242"

Output:

Please wait while the partitions are deleted. This may take several minutes
1 partition has been successfully deleted

Updated 21 days ago