AWS SQS

The Amazon Simple Queue Service (SQS) data connector is implemented for source and sink connections.

The SQS connector supports "First In, First Out" (FIFO) queues only to facilitate the anonymization process.

This is a minimal example of the SQS connector URI scheme:

sqs://queue?region=eu-west-1

📘

Note

The SQS connector for source and sink connections works only with text data formats with encoding types such as "XML" or "JSON".

Authentication

This connector supports two authentication methods:

• Default authentication

  This method is used when credentials aren't provided in the connector URI, for example: sqs://queue?region=... In that case, the default AWS SDK credentials provider (reading environment variables, system properties, ~/.aws/credentials) is used.

• Explicit (static) credentials provider

  This method is used when the credentials are provided explicitly in the connector URI, for example: sqs://aws-access-id:aws-secret-key@queue?region=...

Configuration

Use the following parameters to configure the connector.

The region parameter

Mandatory parameter. Defines the AWS region.

The poll_interval parameter

Optional parameter. Defines the time interval between checks for messages in the source queue. The value must be in the Java Duration format.

Default value: PT0S (0 seconds)

The read_timeout parameter

Optional parameter. Defines the duration for which the call waits for a message to arrive in the queue before returning. If a message is available, the call returns sooner than wait time. If no messages are available and the wait time expires, the call returns successfully with an empty list of messages. The value must be in the Java Duration format.

Default value: PT0.5S (500 milliseconds)

The prefetch_limit parameter

Optional parameter. Defines the number of messages to prefetch. Prefetching works by allowing the message consumer to retrieve a batch of messages from the SQS queue and hold them locally, ready for processing. This helps reduce the overhead caused by fetching messages one by one, improve throughput, and reducing latency. Relevant only when the SQS connector is used as a source connector.

Default value: 1 (1 message)

The api_call_timeout parameter

Optional parameter. Defines the maximum allowed time to complete AWS SQS API calls, including the necessary retires.

To learn more, see the AWS Documentation.

The value must be in the Java Duration format.

The api_call_attempt_timeout parameter

Optional parameter. Defines how long an AWS SQS API call attempt can last before it times out and is retried.

To learn more, see the AWS Documentation.

The value must be in the Java Duration format.

The endpoint parameter

Optional parameter. Enables the use of alternatively hosted SQS API, such as a LocalStack deployment, for example: sqs://queue?region=eu-west-1&endpoint=http://localhost:4566

SQS message group ID

In SQS "First In, First Out" (FIFO) queues, messages are ordered based on the message group ID. For each message group ID, all messages are sent and received in strict order.

HERE Anonymizer Self-Hosted allows all probe data of a single vehicle to be placed into a single message group. This means that there's always one message group that gets data from a specific vehicle.

If the data can be categorized by a vehicle/trajectory/trace identifier or by a geospatial identifier, such as a state or a big enough map tile, all SQS messages must be directed into a single message group that is mapped to this specific identifier.

The easiest way to fulfill this requirement is to have all messages keyed using keys that are equal to category identifiers. For example, all messages for vehicle A must have the key A and all messages from the region California must have the California key.

To learn more, see the AWS documentation.