GuidesPipeline API ReferenceChangelog
Guides
  1. Command Line Interface
  2. OLP CLI Command reference

local catalog layer partition

local catalog layer partition

The OLP CLI supports the following:

  • list partitions in a local layer
  • get partitions from a local layer
  • put partitions in a local layer
  • delete partitions in a local layer

local catalog layer partition list

Lists the names of all partitions in the specified catalog layer; either
versioned, volatile, or index.

olp local catalog layer partition list <catalog HRN> <layer ID> [command options]

Note

The olp catalog layer partition list command output is limited to 100,000 partitions.
If this limit is exceeded, the following warning message appears: "WARNING:
The limit of 100,000 partitions has been reached" The warning message appears only
if the command has been executed without the --json or --quiet flag.

Required parameters:

  • <catalog HRN> The HRN of the catalog.

  • <layer ID> The ID of the layer; either versioned, volatile, or index.
    For stream layers, see local catalog layer stream commands.

  • --filter <filter query> The query string required for Index layers. The
    <filter query> must be written in
    Feed Item Query Language.
    We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators
    instead of the <, <=, >, or >= operators on Windows. All special
    characters inside the <filter query> should be escaped or quoted.

    Note

    --filter <filter query> is optional for versioned and volatile layers,
    where it is used as a prefix to filter partitions by partition name.

Optional parameters:

  • --json Displays the command result in JSON format.
  • --quiet Displays partition names, each on a new line.

Optional parameters for a versioned layer:

  • --version <catalog version> The catalog's metadata version.

Optional parameters for a volatile layer:

  • --modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile
    layers based on when they were last modified. The timestamp must match the
    ISO-8601 format YYYY-MM-DDThh:mm:ss but only YYYY is mandatory. To
    skip values in the timestamp, you must start from the least significant value,
    from right to left.

Example:

The command below lists all partitions from layer1 in the catalog
catalog1.

olp local catalog layer partition list hrn:local:data:::catalog1 layer1

Output:

partition                    size                dataHandle                              checksum                                          CRC
testPartition                23                  415efy1d-5d50-27ge-5876-84a26f2a6a43

Total size: 23 B
Use olp local catalog layer partition get <catalog HRN> <layer ID> to download partitions

local catalog layer partition get

Downloads the partitions that are specified with the --partitions
parameter in the local catalog layer.

olp local catalog layer partition get <catalog HRN> <layer ID> [command options]

Note

Partition(s) will be automatically decompressed during olp local catalog layer partition get
command execution if the layer was created with the --content-encoding <gzip>
parameter.

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to use; either versioned, volatile, or index.
    For stream layers, see local catalog layer stream commands.

Required parameters for an index layer:

  • --filter <filter query> The query string required for Index layers. The
    <filter query> must be written in Feed Item Query Language.
    We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators
    instead of the <, <=, >, or >= operators on Windows. All special
    characters inside the <filter query> should be escaped or quoted. This
    parameter is optional for versioned and volatile layers, where it is used as a
    prefix to filter the partitions by partition name.

Optional parameters:

  • --partitions <partitionID1 partitionID2 ...> The partitions to download.
    If only one partition is specified and no output folder is provided, the
    partition content is retrieved into the standard output. If the partition
    content is retrieved into the standard output, the --modified-since parameter
    is used, and there are no modifications after the date provided, then the
    command execution finishes with the exit code 4.
  • --all Downloads all partitions in the layer.
  • --output <path to dir> The output directory to store the downloaded
    partitions. If the specified directory doesn't exist, it is automatically
    created. If no directory is specified, the default value is
    <current dir>/<catalog HRN>.
  • --quiet Displays the absolute path to the output directory.

Note

For versioned or volatile layers, you must specify either the --all or the
--partitions parameter.

Optional parameters for a versioned layer:

  • --version <catalog version> The catalog's metadata version.
  • --decode <true|false> Determines whether the
    partition content is decoded from protobuf and converted to JSON on-the-fly.
    The default value is false, which means the content is not decoded.
    You can specify the path to a schema on a local file system
    with --decode <true|false:[path/to/schema.zip]>;
    otherwise, the default schema linked to the layer is used.
    An exception is thrown when there is no schema linked to
    the layer or no path to a local schema ZIP provided.
    Schemas can be downloaded from the HERE platform portal or with the
    olp schema get command.
    Using --decode true with --output or --all parameters produces files where
    the following name pattern is used: <partition-name>-decoded.json.

Optional parameters for a volatile layer:

  • --modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile
    layers based on when they were last modified. The timestamp must match the
    ISO-8601 format YYYY-MM-DDThh:mm:ss, but only YYYY is mandatory. To
    skip values in the timestamp, you must start from the least significant value,
    from right to left.
  • --decode <true|false> Determines whether the
    partition content is decoded from protobuf and converted to JSON on-the-fly.
    The default value is false, which means the content is not decoded.
    You can specify the path to a schema on a local file system
    with --decode <true|false:[path/to/schema.zip]>;
    otherwise, the default schema linked to the layer is used.
    An exception is thrown when there is no schema linked to
    the layer or no path to a local schema ZIP provided.
    Schemas can be downloaded from the HERE platform portal or with the
    olp schema get command.
    Using --decode true with --output or --all parameters produces files where
    the following name pattern is used: <partition-name>-decoded.json.

Example:

The command below retrieves the partition content into the standard
output if you specify only one partition.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition1
olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition1

Output:

Local partition content example

Example:

The command below downloads partition1 from layer1 in catalog1 to the specified output file.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition1 > PATH/TO/OUTPUT_FILE
olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition1 > PATH/TO/OUTPUT_FILE

Example:

The command below downloads partition1 and partition2 from version 1
of versioned-layer in catalog1 to the specified output directory.

olp local catalog layer partition get hrn:local:data:::catalog1 versioned-layer \
 --partitions partition1 partition2 --output PATH/TO/OUTPUT_DIRECTORY --version 1
olp local catalog layer partition get hrn:local:data:::catalog1 versioned-layer ^
--partitions partition1 partition2 --output PATH/TO/OUTPUT_DIRECTORY --version 1

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions: partition1,partition2 were successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Example:

To customize the path where the partition is saved, concatenate the
relative or the absolute path with the partition name, using a colon (:).

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --partitions partition_name:PATH/TO/OUTPUT_DIRECTORY
olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--partitions partition_name:PATH/TO/OUTPUT_DIRECTORY

Output:

0% [>                                                  ] 0/1 (0:00:00 / ?)
0% [>                                                  ] 0/1 (0:00:00 / ?)
100% [=============================================] 1/1 (0:00:00 / 0:00:00)

Partition partition_name was successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Example:

The command below retrieves all partitions from the index layer that were uploaded
on January 1, 2020, between 00:01:00 AM and 00:05:00 AM with locationTag equal to 4242.

olp local catalog layer partition get hrn:local:data:::catalog1 layer1 \
 --filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
--output PATH/TO/OUTPUT_DIRECTORY
olp local catalog layer partition get hrn:local:data:::catalog1 layer1 ^
--filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
--output PATH/TO/OUTPUT_DIRECTORY

Output:

0% [>                                                  ] 0/3 (0:00:00 / ?)
0% [>                                                  ] 0/3 (0:00:00 / ?)
0% [>                                                  ] 0/3 (0:00:00 / ?)
100% [=============================================] 3/3 (0:00:00 / 0:00:00)

Partitions: partition1,partition2,partition3 were successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Note

The --filter value depends on the --index-fields value that was specified during
partitions upload (olp local catalog layer partition put command execution). Also, note
that the --index-fields value depends on the --index-definitions value
specified during index layer creation - catalog layer add command execution.

local catalog layer partition put

Publishes the specified files to the specified local partitions.

olp local catalog layer partition put <catalog HRN> <layer ID> [command options]

Note

Partition(s) content will be automatically compressed during the
olp local catalog layer partition put command execution if the layer was created with
the --content-encoding <gzip> parameter.

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to upload partition(s) to; either versioned,
    volatile, or index. For stream layers, see
    local catalog layer stream commands.

Required parameters for an index layer:

  • --index-fields <bool|int|string|heretile|timewindow>:<index-key>:<index-value>
    A list of index fields for this publication. The <timewindow> index field
    definition is required and should be the same as was specified in the layer's
    index definitions.

    If the --index-fields parameter is not passed and the --input parameter is
    passed, the application tries to read the index configuration from the
    index.properties file located in the --input path. The format of the
    index.properties file is the following:

    • fileMask A regular expression used to filter files in the input
      directory.
    • indexType An index type name with these acceptable values: bool,
      int, string, heretile, timewindow. You must define timewindow
      that should be the same as the one specified in the layer
      index-definitions.
    • indexKey An index name. This key is used for listing index partitions.
    • indexValue The value of the index. The value of timewindow should be
      the same as the one specified in the layer index-definitions.

    The following example sets the timewindow index myIndex to 50000 for
    all the files whose names start with file and one digit like file1.bin,
    file2.txt, and the like. Add the boolean index key myBoolIndex set to
    true to the file1.bin file.

    fileMask=file[0-9].*
    indexType=timewindow
    indexKey=myIndex
    indexValue=50000

    fileMask=file1.bin
    indexType=bool
    indexKey=myBoolIndex
    indexValue=true
An empty line between index definitions is not required but it improves the
readability of the config. The values for all configuration keys
(`fileMask`, `indexType`, `indexKey`, `indexValue`) are required.

If both of the `--index-fields` and `--input` parameters are passed, then
the configurations from `index.properties` are ignored.

Optional parameters:

  • --partitions <partition ID>:<path to file> The partitions and files to
    upload.
  • --input <path to dir> The input directory from which to upload the files;
    supports relative paths; hidden files are skipped.
  • --dependency <catalog-hrn:version[:direct]> The list of the dependencies for
    this publication. direct is optional, and its default value is true.
  • --quiet Displays empty output with no additional information.

Note

If you wish to use both the --partitions and --input options, specify the
directory containing the files you wish to put with the --input option.
Then, specify the relative paths to the individual files you wish to put from
that directory using the --partitions parameter.

Optional parameters for a versioned layer:

  • --encode <true|false:[path/to/schema.zip]> Determines whether the
    partition content is encoded from JSON and converted to protobuf on-the-fly.
    The default value is false, which means the content is not encoded.
    You can specify the path to a schema on a local file system;
    otherwise, the default schema linked to the layer is used.
    An exception is thrown when there is no schema linked to
    the layer or no path to a local schema ZIP provided.
    Schemas can be downloaded from the HERE platform portal or with the
    olp schema get command.

Note

You can encode only the JSON file that you got from the olp local catalog layer partition get command.

Optional parameters for a volatile layer:

  • --encode <true|false:[path/to/schema.zip]> Determines whether the
    partition content is encoded from JSON and converted to protobuf on-the-fly.
    The default value is false, which means the content is not encoded.
    You can specify the path to a schema on a local file system;
    otherwise, the default schema linked to the layer is used.
    An exception is thrown when there is no schema linked to
    the layer or no path to a local schema ZIP provided.
    Schemas can be downloaded from the HERE platform portal or with the
    olp schema get command.

Note

You can encode only the JSON file that you got from the olp local catalog layer partition get command.

Optional parameters for an index layer:

  • --metadata <key1=val1 key2=val2 ...> List of all user-defined fields that
    can store extra information. If the --input parameter is provided, the
    metadata applies to all the files located in the input directory.

Example:

The command below uploads file1 and file2 to partition1 and partition2
respectively from the folder relative to the current folder.

olp local catalog layer partition put hrn:local:data:::catalog1 layer1 \
 --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
olp local catalog layer partition put hrn:local:data:::catalog1 layer1 ^
--partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions partition1, partition2 were successfully uploaded

Example:

If you specify only one partition, the command retrieves the partition content
into the standard output. For more information, see
olp local catalog layer partition get.

OLP CLI lets you chain both get and put commands when you retrieve
content into the standard output. This means that you can omit the path to
the partition and use a hyphen (-) instead.

olp local catalog layer partition get hrn:local:data:::catalog-1 versioned-layer1 \
 --partitions partition1 | olp local catalog layer partition put hrn:local:data:::catalog-2 \
 versioned-layer1 --partitions partition1:-
olp local catalog layer partition get hrn:local:data:::catalog-1 versioned-layer1 ^
--partitions partition1 | olp local catalog layer partition put hrn:local:data:::catalog-2 ^
versioned-layer1 --partitions partition1:-

Output:

0% [>                                                  ] 0/1 (0:00:00 / ?)
0% [>                                                  ] 0/1 (0:00:00 / ?)
100% [=============================================] 1/1 (0:00:00 / 0:00:00)

Partition partition1 was successfully uploaded

Example:

The command below uploads file1 and file2 to partition1 and partition2
respectively to the index layer with the --index-fields parameter equal to
timewindow:ingestionTime:1577836960 int:locationTag:4242

Note

The --index-fields value depends on the --index-definitions value
specified during index layer creation - catalog layer add command execution.

olp local catalog layer partition put hrn:local:data:::catalog1 layer1 \
 --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
--index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242
olp local catalog layer partition put hrn:local:data:::catalog1 layer1 ^
--partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
--index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions partition1, partition2 were successfully uploaded

local catalog layer partition delete

Deletes partitions from an index layer by the --filter query. Note that this
command does not support data deletion from versioned and volatile layers.

olp local catalog layer partition delete <catalog HRN> <layer ID> [command options]

Required parameters:

  • <catalog HRN> The HRN of the catalog.
  • <layer ID> The ID of the layer to delete partitions from.
  • --filter <filter query> The query string required to query data that should
    be deleted from the index layer. The <filter query> value must be written in
    the Feed Item Query Language.
    We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators
    instead of the <, <=, >, or >= operators on Windows. All special
    characters inside <filter query> should be escaped or quoted.

Optional parameters:

  • --timeout <duration in seconds> The timeout duration of the delete request in
    seconds. If not specified, the default is 3600.
  • --quiet Displays empty output with no additional information.

Example:

The command below deletes partitions with keys testIndex==3600000 and
env==4242.

olp local catalog layer partition delete hrn:local:data:::catalog1 indexLayer1 \
 --filter "testIndex==3600000;env==4242"
olp local catalog layer partition delete hrn:local:data:::catalog1 indexLayer1 ^
--filter "testIndex==3600000;env==4242"

Output:

Please wait while the partitions are deleted. This may take several minutes
1 partition has been successfully deleted