Azure Blob Storage

The Azure Blob Storage connector is designed for use with the Azure Blob Storage.

It supports 2 operational modes:

• for processing of batch data stored in Azure Blob Storage
• for runtime configuration using anonymization management queue

Batch data processing

This is a minimal example of an Azure Blob Storage connector URI for processing batch data:

az+blob+batch+files://AccountName:AccountKey@ContainerName/path?poll_interval=5&endpoint=https://AccountName.blob.core.windows.net/

When the batch data processing is configured to use the example URI as a source, the HERE Anonymizer Self-Hosted reads the available files according to the defined filter from the given path in the given container.

When the batch data processing is configured to use the example URI as a sink, HERE Anonymizer Self-Hosted writes the anonymized data as files to that location.

Please see explicit authentication and optional parameters in the chapters below.

Runtime configuration

This is a minimal example of an Azure Blob Storage connector URI for runtime configuration:

az+blob+management://AccountName:AccountKey@ContainerName?poll_interval=5&endpoint=https://AccountName.blob.core.windows.net/

When the anonymization management queue is configured to use the example URI, HERE Anonymizer Self-Hosted reads the following predefined blob objects in one-minute intervals:

• https://AccountName.blob.core.windows.net/myContainer/anonymization.conf
• https://AccountName.blob.core.windows.net/myContainer/HERE_ANONYMIZER_LICENSE

Please see explicit authentication and optional parameters in the chapters below.

Logging

HERE Anonymizer Self-Hosted logs one of the following messages, depending on the state of the objects:

• Both objects recognized, detected changes to objects

  INFO  c.h.a.f.c.azure.blob.AzureBlobStorageSourceConnector - Configured blob source of https://AccountName.blob.core.windows.net/myContainer/ reading every PT1M all the blob: [HERE_ANONYMIZER_LICENSE,anonymization.conf]
  ...
  INFO  c.h.a.f.c.azure.blob.AzureBlobStorageSourceFunction - Blob HERE_ANONYMIZER_LICENSE has been changed, downloading new version
  INFO  c.h.a.f.c.azure.blob.AzureBlobStorageSourceFunction - Blob anonymization.conf has been changed, downloading new version

• Objects detected, no changes to objects (checked by Azure Blob's eTag) - the system doesn't log any messages.

Authentication

This connector supports following authentication using explicit (static) credentials provider.

This method requires the credentials being provided explicitly in the connector URI, for example: az+blob+management://AccountName:AccountKey@ContainerName?poll_interval=5&endpoint=https://AccountName.blob.core.windows.net/ The same pattern applies for az+blob+batch+files prefix as well.

Configuration

Use the following parameters to configure the connector.

The AccountName parameter

Mandatory parameter. Defines the account name.

The AccountKey parameter

Mandatory parameter. Defines the account key which is the secret key for the account.

The AccountKey value must be a Base64-encoded string.

The ContainerName parameter

Mandatory parameter. Defines the Azure Blob Storage.

The poll_interval parameter

Optional parameter. Defines the time interval for checking for changes in the configured files.

The value must be in the Java Duration format.

This parameter is applicable for runtime configuration mode only.

Default value: PT1M (1 minute)

The endpoint parameter

Mandatory parameter. Defines the Azure Blob Storage URL that follows the https://AccountName.blob.core.windows.net/ format, or the custom domain name URL, for example https://myblob.example.com.

The blob_names parameter

Optional parameter. Comma-separated list of files to read. Enables the use of alternative filenames for the managed objects (configuration and license).

Default value: anonymization.conf,HERE_ANONYMIZER_LICENSE With the parameter configured to az+blob+management://AccountName:AccountKey@ContainerName?blob_names=new-config-name.conf,CUSTOM_LICENSE_NAME&poll_interval=5&endpoint=https://AccountName.blob.core.windows.net/ the connector observes these Azure Blobs:

• https://AccountName.blob.core.windows.net/ContainerName/new-config-name.conf
• https://AccountName.blob.core.windows.net/ContainerName/CUSTOM_LICENSE_NAME

The path_resolver_recursive parameter

Optional parameter. Defines if the S3 keys should be recursively traversed in a directory-like way with / character as a delimiter.

This parameter is applicable for batch data processing mode only.

Default value: true

The path_resolver_filter parameter

Optional parameter. Defines the PathMatcher pattern which should be applied as a filter to the overall S3 object list relative to the given bucket and path of URI.

This parameter is applicable for batch data processing mode only.

Default value: glob:*

The req_timeout_secs parameter

Optional parameter. Defines the request timeout in seconds when listing blob objects.

This parameter is applicable for batch data processing mode only.

Default value: 30