How to select a streaming connector: Kafka or HTTP

Stream layers are implemented as Kafka clusters.

To read from or write to a stream layer, your application must use one of the
following connector types:

• Direct Kafka is the preferred connector type since it directly communicates
  with the underlying Kafka cluster. It is the default connector.
• HTTP Connector is an HTTP wrapper on top of Kafka, and therefore implies a
  communication overhead.

If required, you can provide a different connector type for the producer and the
consumer. For example, your application can use Direct Kafka to read from a
stream layer and the HTTP connector to write to a different stream layer.

Direct Kafka

To use the Direct Kafka connector type, you must provide the following
dependency:

<dependencies>
    <dependency>
        <groupId>com.here.platform.data.client</groupId>
        <artifactId>kafka-support_2.13</artifactId>
        <version>1.22.61</version>
    </dependency>
</dependencies>

libraryDependencies += "com.here.platform.data.client" %% "kafka-support" % "1.22.61"

Note that, if your application uses flink-support, you do not need to add this
dependency as the dependency is already included in flink-support. For more
details, refer to the
Dependencies for Stream Pipelines section.

To enable Direct Kafka support, you must provide the following configuration:

here.platform.data-client.stream.connector {
  producer = "kafka-connector"
  consumer = "kafka-connector"
}

Producers and consumers can be configured independently. For more information,
see the Data Client configuration file in the
Configuration section.

HTTP connector

If you cannot use Kafka Direct, for example because your application is running
behind a proxy, use the HTTP connector.

If you cannot use Kafka Direct, for example because your application is running
behind a proxy, use the HTTP connector. To use it, the following configuration
must be set:

here.platform.data-client.stream.connector {
  producer = "http-connector"
  consumer = "http-connector"
}

Producers and consumers can be configured independently. For more information,
see the Data Client configuration file in the
Configuration section.

Note

Kafka consumer group name
If your application uses the HTTP Connector type, the connector will define a
unique
Kafka consumer group name
for a given combination of credentials, catalog HRN and layer ID. As a result,
if you run two applications that use the same credentials, and consume data
from the same stream layer, any given message will be consumed by only one
application. Specify a unique kafka consumer group in each application to
avoid this behavior.