HERE platform stream layer

The HERE platform stream layer data connector is implemented for source and sink connections.

This is the HERE platform stream layer data connector URI scheme:

olp+stream://platform.here.com/hrn:here:data::my-org:my-catalog/my-stream-layer

The source connector supports these query parameters:

consumer_group: arbitrary string
offset: latest or earliest values

The source and sink connectors both support this optional query parameter:

creds_file: path to the custom credentials file

This is an example of a source URI:

olp+stream://platform.here.com/hrn:here:data::my-org:my-catalog/my-stream-layer?consumer_group=MyGroup&offset=earliest&creds_file={CREDENTIALS_FILE_PATH}

Format auto-detection

The HERE platform stream layer connector supports format auto-detection. The connector recognizes the input and output data format automatically, which means that you can set SOURCE_FORMAT and SINK_FORMAT to auto or leave the value empty.

The conditions for successful auto-detection are:

The source and sink layers must have a schema assigned.
Assigned schema name must contain sensoris, sdii_message_list or sdii_message.

Apache Kafka notice

Note that the HERE platform stream layer data connector uses an Apache Kafka backend.

Kafka doesn't keep the order of incoming messages if parallel processing is enabled and only provides a total order of messages in a partition, not between different partitions in a topic.

If keeping the order of incoming messages is important for your use case, don't enable parallel processing. To ensure that parallel processing is disabled, set STREAM PARTITION PARALLELIZATION (PARTITIONS) to 1 when you create a stream layer.