Azure Event Hubs

The Azure Event Hubs data connector is implemented for source and sink connections to stream data using the AMQP protocol. Since the Azure Event Hubs support Apache Kafka, you can also use the Apache Kafka connector to connect with the event hubs.

To learn more, see the Kafka connector documentation.

This is a minimal example of the Event Hubs connector URI scheme:

evh://namespace/event-hub

Authentication

This connector supports two authentication methods:

• Default authentication When credentials aren't specified, the system uses the default Azure credentials provider.

  For example: evh://namespace/event-hub...

• Explicit (static) credentials provider Uses credentials provided explicitly to the URI. For example: evh://SharedAccessKeyName:SharedAccessKey@namespace/event-hub...

Configuration

Use the following parameters to configure the connector.

The consumer_group parameter

Optional parameter. A consumer group is a view of the entire Event Hub. Consumer groups allow multiple apps connected to the hub to:

• Have separate views of the event stream.
• Read the event stream independently.
• Read the event stream at app-specific pace and from app-specific position.

The maximum number of concurrent readers on a partition per consumer group is 5. It's recommended that there's only one active consumer for a given partition and consumer group pairing.

Each active reader receives the events from its partition. If there are multiple readers on the same partition, they receive duplicate events.

The default value is $Default. To learn more, see the Consumer groups documentation

The offset parameter

Optional parameter. Defines the position within the Event Hub partition to begin consuming events.

Supported values: earliest and latest.

Default value: latest.

The partitions parameter

Optional parameter. Configures a list of partitions to be read. Event Hubs organize the received data into one or more partitions. For an event hub with four partitions, partition IDs 0, 1, 2, and 3 are valid.

Relevant only when the connector is used as a source connector. By default, the connector consumes all partitions of the topic.

The max_wait_time parameter

Optional parameter. Relevant only when the connector is used as a source connector. Value defined using the Java Duration format.

Defines the maximum amount of time the system waits for an event to be available when reading before it reads an empty event. If a message is available, the system returns the call without waiting for the time defined in WaitTimeSeconds.

If no messages are available and the wait time expires, the call returns successfully with an empty event.

To learn more, see the EventHubConsumerClient Class documentation.

Examples:

• PT1M: 1 minute
• PT1H: 1 hour
• PT30S: 30 seconds
• PT23H56M4.091S: the sidereal day

The default value is PT0.5S.

The max_message_count parameter

Optional parameter. Relevant only when the connector is used as a source connector.

Defines the maximum number of messages to receive in a batch.

To learn more, see the EventHubConsumerClient Class.

The default value is 1.

The prefetch_count parameter

Optional parameter. Relevant only when the connector is used as a source connector.

The number of events that will be eagerly requested from the Event Hubs service and queued locally regardless of whether a read operation is currently taking place, to increase maximize throughput by allowing events tobe read from a local cache rather than waiting for a service request.

To learn more, see the EventHubClientBuilder Class documentation.

Using multi-partitioned sources

HERE Anonymizer Self-Hosted allows all probe data of a single vehicle to be placed into a single dedicated partition of a multi-partitioned event hub. This means that there's always one partition that gets data from a specific vehicle.

If the data can be categorized by a vehicle/trajectory/trace identifier or by a geospatial identifier, such as a state or a big enough map tile, all the messages must be directed into a single partition that is mapped to this specific identifier.

The easiest way to fulfil this requirement is to have all messages keyed using keys that are equal to category identifiers. For example, all messages for vehicle A must have the key A and all messages from the region California must have the California key.

To learn more, see Features and terminology in Azure Event Hubs