local catalog layer stream

The OLP CLI supports the following:

• get partitions from a local stream layer
• put partitions to a local stream layer

local catalog layer stream get

Subscribes to the specified local stream layer and downloads or prints every partition that is uploaded to the layer during the subscription period.

olp local catalog layer stream get <catalog HRN> <layer ID> [command options]

Required parameters:

• <catalog HRN> The HRN of the catalog.
• <layer ID> The ID of the layer.

Optional parameters:

• --output <path to dir> The output directory to store data received from the stream. Files are created with the names of partitions received from the stream. If the specified directory doesn't exist, it is automatically created. If no directory is specified, partitions from the stream are printed to the standard output separated by a delimiter.

  Note

  When a partition with a specific name is downloaded multiple times, new files are created using the following naming pattern:

  • partitionName
  • partitionName_1
  • partitionName_2

• --delimiter <character> A delimiter used for printing stream data to the standard output; empty by default.

• --limit <max number of partitions> The maximum number of partitions to be downloaded.

• --timeout <duration in seconds> The subscription timeout duration in seconds (infinite by default).

• --consumer-group <consumer ID> The name of a consumer, or a group if the consumer is distributed.

• --decode <true|false> Determines whether the partition content is decoded from protobuf and converted to JSON on-the-fly. The default value is false, which means the content is not decoded. You can specify the path to a schema on a local file system with --decode <true|false:[path/to/schema.zip]>; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command. Using --decode true with --output parameter produces files where the following name pattern is used: <partition-name>-decoded.json.

• --quiet Displays the absolute path to the output directory, if available. Otherwise, the partitions' contents are printed from the stream to the standard output.

• --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.

• --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below prints all partitions from stream-layer1 in the catalog1 using the default empty delimiter.

olp local catalog layer stream get hrn:local:data:::catalog1 stream-layer1

Output:

Local partition content example

local catalog layer stream put

Publishes the content from the filesystem into a local catalog's stream layer.

olp local catalog layer stream put <catalog HRN> <layer ID> [command options]

Required parameters:

• <catalog HRN> The HRN of the catalog.
• <layer ID> The ID of the layer.

Optional parameters:

• --input <path to dir> The input directory from where to read partitions. Relative paths are supported. Partitions are populated by walking the file tree rooted at the given input directory. The file tree is traversed depth-first and in lexicographical order. If not specified, the partitions are taken from the standard input and split by the delimiter.
• --delimiter <character> A delimiter used for reading partitions from the standard input.
• --limit <max number of partitions> The maximum number of partitions to be published.
• --encode <true|false:[path/to/schema.zip]> Determines whether the partition content is encoded from JSON and converted to protobuf on-the-fly. The default value is false, which means the content is not encoded. Supported layer content types: protobuf. You can specify the path to a schema on a local file system; otherwise, the default schema linked to the layer is used. An exception is thrown when there is no schema linked to the layer or no path to a local schema ZIP provided. Schemas can be downloaded from the HERE platform portal or with the olp schema get command.

Note

You can encode only the JSON file that you got from the olp local catalog layer stream get command.

• --credentials <path to credentials file> The name of a credentials file to use with the command. Credentials files are downloaded separately from the platform portal.
• --profile <profile name> The name of the credentials profile to use from the olpcli.ini file.
• --quiet Displays empty output with no additional information.

For more information on using credentials and profiles, see Credentials setup.

Example:

The command below publishes all partitions from /path/to/directory.

olp local catalog layer stream put hrn:local:data:::catalog1 stream-layer1 --input /path/to/directory

Output:

Publishing of partitions to stream-layer1 layer completed