catalog layer partition

The OLP CLI supports the following:

list partitions in a layer
get partitions from a layer
put partitions in a layer
delete partitions in a layer

catalog layer partition list

Lists the names of all partitions in the specified catalog layer; either
versioned, volatile, or index.

olp catalog layer partition list <catalog HRN> <layer ID> [command options]

Note

The olp catalog layer partition list command output is limited to 100,000 partitions.
If this limit is exceeded, the following warning message appears: "WARNING:
The limit of 100,000 partitions has been reached" The warning message appears only
if the command has been executed without the --json or --quiet flag.

Required parameters:

<catalog HRN> The HRN of the catalog.
<layer ID> The ID of the layer; either versioned, volatile, or index.
For stream layers, see Stream commands
--filter <filter query> The query string required for Index layers. The
<filter query> must be written in
Feed Item Query Language.
We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators
instead of the <, <=, >, or >= operators on Windows. All special
characters inside the <filter query> should be escaped or quoted.

Note

--filter <filter query> is optional for versioned and volatile layers,
where it is used as a prefix to filter partitions by partition name.

Optional parameters:

--credentials <path to credentials file> The name of a credentials file to use
with the command. Credentials files are downloaded separately from the platform portal.
--profile <profile name> The name of the credentials profile to use from the
olpcli.ini file.
--json Displays the command result in JSON format.
--quiet Displays partition names, each on a new line.
--scope <project HRN> Specifies the project HRN to use as the scope in the
request. The value specified with --scope overrides any value for
here.token.scope provided in the credentials file used for the command.

Optional parameters for a versioned layer:

--version <catalog version> The catalog's metadata version.
--change startVersion:[endVersion] Get metadata for the latest partition changes for a version range in a versioned layer. Only the latest change for each changed partition is returned, not intermediate changes.
If the response contains a partition with an empty dataHandle value, it means that the partition was deleted.
If a partition was created and deleted between the versions specified in the startVersion and endVersion parameters, the partition is included in the response with an empty dataHandle value. For example,
if you request versions 0 to 10, and a partition was created in version 3 and then deleted in version 7, it is included in the response with an empty dataHandle value.
If endVersion is not specified, the latest version of the catalog metadata will be used.
--context <default|super|extension> The context where the operation will be performed on composite layers.
Available context are:
- default For composite layers, the operation is executed for both the current layer and the layer it extends. For normal layers, this is the only valid context.
- extension The operation will be executed only in the extension, and no operation will be performed in the extended layer.
- super The operation will be executed only in the layer being extended.

Optional parameters for a volatile layer:

--modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile
layers based on when they were last modified. The timestamp must match the
ISO-8601 format YYYY-MM-DDThh:mm:ss but only YYYY is mandatory. To
skip values in the timestamp, you must start from the least significant value,
from right to left.

For more information on using credentials and profiles, see
Credentials setup.

Example:

The command below lists all partitions from layer1 in the catalog
catalog1.

olp catalog layer partition list hrn:here:data::org:catalog1 layer1

Output:

partition                    size                dataHandle                              checksum                                          CRC
testPartition                23                  3f24f359-4b6d-4d51-a39f-1140r112w111

Total size: 23 B
Use olp catalog layer partition get <catalog HRN> <layer ID> to download partitions

catalog layer partition get

Downloads the partitions that are specified with the --partitions
parameter in the catalog layer.

olp catalog layer partition get <catalog HRN> <layer ID> [command options]

Note

Partition(s) will be automatically decompressed during olp catalog layer partition get
command execution if the layer was created with the --content-encoding <gzip>
parameter.

Required parameters:

<catalog HRN> The HRN of the catalog.
<layer ID> The ID of the layer to use; either versioned, volatile, or index.
For stream layers, see catalog layer stream commands.

Required parameters for an index layer:

--filter <filter query> The query string required for Index layers. The
<filter query> must be written in Feed Item Query Language.
We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators
instead of the <, <=, >, or >= operators on Windows. All special
characters inside the <filter query> should be escaped or quoted. This
parameter is optional for versioned and volatile layers, where it is used as a
prefix to filter the partitions by partition name.

Optional parameters:

--partitions <partitionID1 partitionID2 ...> The partitions to download.
If only one partition is specified and no output folder is provided, the
partition content is retrieved into the standard output. If the partition
content is retrieved into the standard output, the --modified-since parameter
is used, and there are no modifications after the date provided, then the
command execution finishes with the exit code 4.

Note

The command can produce empty output when trying to write the content of the partition
that contains non-ASCII symbols to stdout. This is a Windows-specific problem, related to
the peculiarity of rendering UTF-8 symbols on Windows console. As a workaround, please redirect
command output to a file or specify the output directory with the --output parameter.
--all Downloads all partitions in the layer.
--parallelism <number> Configure the number of concurrent download requests.

Note

When processing large numbers of partitions, command now automatically switches to PERFORMANCE client
configuration mode, which optimizes HTTP connection pooling and allows more open connections.
--output <path to dir> The output directory to store the downloaded
partitions. If the specified directory doesn't exist, it is automatically
created. If no directory is specified, the default value is
<current dir>/<catalog HRN>.
--credentials <path to credentials file> The name of a credentials file to use
with the command. Credentials files are downloaded separately from the platform portal.
--profile <profile name> The name of the credentials profile to use from the
olpcli.ini file.
--quiet Displays the absolute path to the output directory.
--scope <project HRN> Specifies the project HRN to use as the scope in the
request. The value specified with --scope overrides any value for
here.token.scope provided in the credentials file used for the command.

Note

For versioned or volatile layers, you must specify either the --all or the
--partitions parameter.

Optional parameters for a versioned layer:

--version <catalog version> The catalog's metadata version.
--decode <true|false> Determines whether the
partition content is decoded from protobuf and converted to JSON on-the-fly.
The default value is false, which means the content is not decoded.
You can specify the path to a schema on a local file system
with --decode <true|false:[path/to/schema.zip]>;
otherwise, the default schema linked to the layer is used.
An exception is thrown when there is no schema linked to
the layer or no path to a local schema ZIP provided.
Schemas can be downloaded from the HERE platform portal or with the
olp schema get command.
Using --decode true with --output or --all parameters produces files where
the following name pattern is used: <partition-name>-decoded.json.
--context <default|super|extension> The context where the operation will be performed on composite layers.
Available context are:
- default For composite layers, the operation is executed for both the current layer and the layer it extends. For normal layers, this is the only valid context.
- extension The operation will be executed only in the extension, and no operation will be performed in the extended layer.
- super The operation will be executed only in the layer being extended.

Optional parameters for a volatile layer:

--modified-since <YYYY-MM-DDThh:mm:ss> Filters the partitions for volatile
layers based on when they were last modified. The timestamp must match the
ISO-8601 format YYYY-MM-DDThh:mm:ss, but only YYYY is mandatory. To
skip values in the timestamp, you must start from the least significant value,
from right to left.
--decode <true|false> Determines whether the
partition content is decoded from protobuf and converted to JSON on-the-fly.
The default value is false, which means the content is not decoded.
You can specify the path to a schema on a local file system
with --decode <true|false:[path/to/schema.zip]>;
otherwise, the default schema linked to the layer is used.
An exception is thrown when there is no schema linked to
the layer or no path to a local schema ZIP provided.
Schemas can be downloaded from the HERE platform portal or with the
olp schema get command.
Using --decode true with --output or --all parameters produces files where
the following name pattern is used: <partition-name>-decoded.json.

For more information on using credentials and profiles, see
Credentials setup.

Example:

The command below retrieves the partition content into the standard
output if you specify only one partition.

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 \
   --partitions partition1

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 ^
   --partitions partition1

Output:

Partition content example

Example:

The command below downloads partition1 from layer1 in catalog1 to the specified output file.

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 \
   --partitions partition1 > PATH/TO/OUTPUT_FILE

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 ^
   --partitions partition1 > PATH/TO/OUTPUT_FILE

Example:

The command below downloads partition1 and partition2 from version 1
of versioned-layer in catalog1 to the specified output directory.

olp catalog layer partition get hrn:here:data::org:catalog1 versioned-layer \
   --partitions partition1 partition2 --output PATH/TO/OUTPUT_DIRECTORY --version 1

olp catalog layer partition get hrn:here:data::org:catalog1 versioned-layer ^
   --partitions partition1 partition2 --output PATH/TO/OUTPUT_DIRECTORY --version 1

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions: partition1,partition2 were successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Example:

To customize the path where the partition is saved, concatenate the
relative or the absolute path with the partition name, using a colon (:).

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 \
   --partitions partition_name:PATH/TO/OUTPUT_DIRECTORY

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 ^
   --partitions partition_name:PATH/TO/OUTPUT_DIRECTORY

Output:

0% [>                                                  ] 0/1 (0:00:00 / ?)
0% [>                                                  ] 0/1 (0:00:00 / ?)
100% [=============================================] 1/1 (0:00:00 / 0:00:00)

Partition partition_name was successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Example:

The command below retrieves all partitions from the index layer that were uploaded
on January 1, 2020, between 00:01:00 AM and 00:05:00 AM with locationTag equal to 4242.

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 \
   --filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
   --output PATH/TO/OUTPUT_DIRECTORY

olp catalog layer partition get hrn:here:data::org:catalog1 layer1 ^
   --filter "ingestionTime>1577836860;ingestionTime<1577837100;locationTag==4242"
   --output PATH/TO/OUTPUT_DIRECTORY

Output:

0% [>                                                  ] 0/3 (0:00:00 / ?)
0% [>                                                  ] 0/3 (0:00:00 / ?)
0% [>                                                  ] 0/3 (0:00:00 / ?)
100% [=============================================] 3/3 (0:00:00 / 0:00:00)

Partitions: partition1,partition2,partition3 were successfully downloaded to PATH/TO/OUTPUT_DIRECTORY

Note

The --filter value depends on the --index-fields value that was specified during
partitions upload (olp catalog layer partition put command execution). Also, note
that the --index-fields value depends on the --index-definitions value
specified during index layer creation - catalog layer add command execution.

Query Performance Tips

Use the timewindow indexing attribute in the query.
To get results back within a short period of time, you should split
queries based on timewindow so that the data in the timewindow range
is manageable (100,000 or less).
Limit the usage of additional constraints.

catalog layer partition put

Publishes the specified files to the specified partitions.

olp catalog layer partition put <catalog HRN> <layer ID> [command options]

Note

Partition(s) content will be automatically compressed during the
olp catalog layer partition put command execution if the layer was created with
the --content-encoding <gzip> parameter.

Required parameters:

<catalog HRN> The HRN of the catalog.
<layer ID> The ID of the layer to upload partition(s) to; either versioned,
volatile, or index. For stream layers, see
catalog layer stream commands.

Required parameters for an index layer:

--index-fields <bool|int|string|heretile|timewindow>:<index-key>:<index-value>
A list of index fields for this publication. The <timewindow> index field
definition is required and should be the same as was specified in the layer's
index definitions.

If the --index-fields parameter is not passed and the --input parameter is
passed, the application tries to read the index configuration from the
index.properties file located in the --input path. The format of the
index.properties file is the following:
- fileMask A regular expression used to filter files in the input
  directory.
- indexType An index type name with these acceptable values: bool,
  int, string, heretile, timewindow. You must define timewindow
  that should be the same as the one specified in the layer
  index-definitions.
- indexKey An index name. This key is used for listing index partitions.
- indexValue The value of the index. The value of timewindow should be
  the same as the one specified in the layer index-definitions.
The following example sets the timewindow index myIndex to 50000 for
all the files whose names start with file and one digit like file1.bin,
file2.txt, and the like. Add the boolean index key myBoolIndex set to
true to the file1.bin file.

    fileMask=file[0-9].*
    indexType=timewindow
    indexKey=myIndex
    indexValue=50000

    fileMask=file1.bin
    indexType=bool
    indexKey=myBoolIndex
    indexValue=true

An empty line between index definitions is not required but it improves the
readability of the config. The values for all configuration keys
(`fileMask`, `indexType`, `indexKey`, `indexValue`) are required.

If both of the `--index-fields` and `--input` parameters are passed, then
the configurations from `index.properties` are ignored.

Optional parameters:

--partitions <partition ID>:<path to file> The partitions and files to
upload.
--parallelism <number> Configure the number of concurrent upload requests.

Note

When processing large numbers of partitions, command now automatically switches to PERFORMANCE client
configuration mode, which optimizes HTTP connection pooling and allows more open connections.
--input <path to dir>/<glob-pattern> The input directory, single file, or filename glob pattern (non-recursive)
from which to upload the files; supports relative paths; hidden files are skipped.

Note

If neither --input nor --partitions parameters are specified, nothing will be uploaded, but metadataVersion will be incremented.

If you wish to use both the --partitions and --input options, specify the
directory containing the files you wish to put with the --input option.
Then, specify the relative paths to the individual files you wish to put from
that directory using the --partitions parameter.

--dependency <catalog-hrn:version[:direct]> List of the dependencies for this publication
where direct is either true or false, and its default value is true.
--credentials <path to credentials file> The name of a credentials file to use
with the command. Credentials files are downloaded separately from the platform portal.
--profile <profile name> The name of the credentials profile to use from the
olpcli.ini file.
--quiet Displays empty output with no additional information.
--scope <project HRN> Specifies the project HRN to use as the scope in the
request. The value specified with --scope overrides any value for
here.token.scope provided in the credentials file used for the command.

For more information on using credentials and profiles, see
Credentials setup.

Optional parameters for a versioned layer:

--encode <true|false:[path/to/schema.zip]> Determines whether the
partition content is encoded from JSON and converted to protobuf on-the-fly.
The default value is false, which means the content is not encoded.
You can specify the path to a schema on a local file system;
otherwise, the default schema linked to the layer is used.
An exception is thrown when there is no schema linked to
the layer or no path to a local schema ZIP provided.
Schemas can be downloaded from the HERE platform portal or with the
olp schema get command.

Note

You can encode only the JSON file that you got from the olp catalog layer partition get command.

Optional parameters for a volatile layer:

--encode <true|false:[path/to/schema.zip]> Determines whether the
partition content is encoded from JSON and converted to protobuf on-the-fly.
The default value is false, which means the content is not encoded.
You can specify the path to a schema on a local file system;
otherwise, the default schema linked to the layer is used.
An exception is thrown when there is no schema linked to
the layer or no path to a local schema ZIP provided.
Schemas can be downloaded from the HERE platform portal or with the
olp schema get command.

Note

You can encode only the JSON file that you got from the olp catalog layer partition get command.

Optional parameters for an index layer:

--metadata <key1=val1 key2=val2 ...> List of all user-defined fields that
can store extra information. If the --input parameter is provided, the
metadata applies to all the files located in the input directory.

Example:

The command below uploads file1 and file2 to partition1 and partition2
respectively from the folder relative to the current folder.

olp catalog layer partition put hrn:here:data::org:catalog1 layer1 \
   --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2

olp catalog layer partition put hrn:here:data::org:catalog1 layer1 ^
   --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions partition1, partition2 were successfully uploaded

Example:

If you specify only one partition, the command retrieves the partition content
into the standard output. For more information, see
olp catalog layer partition get.

OLP CLI lets you chain both get and put commands when you retrieve
content into the standard output. This means that you can omit the path to
the partition and use a hyphen (-) instead.

olp catalog layer partition get hrn:here:data::org:catalog-1 versioned-layer1 \
   --partitions partition1 | olp catalog layer partition put hrn:here:data::org:catalog-2 \
   versioned-layer1 --partitions partition1:-

olp catalog layer partition get hrn:here:data::org:catalog-1 versioned-layer1 ^
   --partitions partition1 | olp catalog layer partition put hrn:here:data::org:catalog-2 ^
   versioned-layer1 --partitions partition1:-

Output:

0% [>                                                  ] 0/1 (0:00:00 / ?)
0% [>                                                  ] 0/1 (0:00:00 / ?)
100% [=============================================] 1/1 (0:00:00 / 0:00:00)

Partition partition1 was successfully uploaded

Example:

The command below uploads file1 and file2 to partition1 and partition2
respectively to the index layer with the --index-fields parameter equal to
timewindow:ingestionTime:1577836960 int:locationTag:4242

Note

The --index-fields value depends on the --index-definitions value
specified during index layer creation - olp catalog layer add command execution.

olp catalog layer partition put hrn:here:data::org:catalog1 layer1 \
   --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
   --index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242

olp catalog layer partition put hrn:here:data::org:catalog1 layer1 ^
   --partitions partition1:PATH/TO/FILE1 partition2:PATH/TO/FILE2
   --index-fields timewindow:ingestionTime:1577836960 int:locationTag:4242

Output:

0% [>                                                  ] 0/2 (0:00:00 / ?)
0% [>                                                  ] 0/2 (0:00:00 / ?)
100% [=============================================] 2/2 (0:00:00 / 0:00:00)

Partitions partition1, partition2 were successfully uploaded

catalog layer partition delete

Deletes partitions from an index layer by the --filter query. Note that this
command does not support data deletion from versioned and volatile layers.

olp catalog layer partition delete <catalog HRN> <layer ID> [command options]

Required parameters:

<catalog HRN> The HRN of the catalog.
<layer ID> The ID of the layer to delete partitions from.
--filter <filter query> The query string required to query data that should
be deleted from the index layer. The <filter query> value must be written in
the Feed Item Query Language.
We recommend using the =lt=, =le=, =gt=, or =ge= comparison operators
instead of the <, <=, >, or >= operators on Windows. All special
characters inside <filter query> should be escaped or quoted.

Optional parameters:

--timeout <duration in seconds> The timeout duration of the delete request in
seconds. If not specified, the default is 3600.
--credentials <path to credentials file> The name of a credentials file to use
with the command. Credentials files are downloaded separately from the platform portal.
--profile <profile name> The name of the credentials profile to use from the
olpcli.ini file.
--quiet Displays empty output with no additional information.

Example:

The command below deletes partitions with keys testIndex==3600000 and
env==4242.

olp catalog layer partition delete hrn:here:data::org:catalog1 indexLayer1 \
   --filter "testIndex==3600000;env==4242"

olp catalog layer partition delete hrn:here:data::org:catalog1 indexLayer1 ^
   --filter "testIndex==3600000;env==4242"

Output:

Please wait while the partitions are being deleted. This may take several minutes
1 partition has been successfully deleted

For more information on using credentials and profiles, see
Credentials setup.