AWS S3 Parquet

The AWS S3 Parquet connector reads batch data from Parquet files stored in Amazon Simple Storage Service (S3). It can also write anonymized batch output to S3 objects without converting the output to Parquet.

It supports two batch operational modes:

• Reading batch data from Parquet files in S3
• Writing anonymized batch data to S3 objects

Source usage

This is a minimal example of an S3 Parquet connector URI for reading batch data:

s3+parquet+batch+files://bucket/path?region=eu-west-1

When you use this URI as a source, HERE Anonymizer Self-Hosted reads matching S3 objects from the configured bucket and path.

The connector processes each matching Parquet file by row group. Each row group becomes a separate batch split and is delivered downstream as a source message body in Parquet format.

Sink usage

For sink usage, use the S3 batch files URI:

s3+batch+files://bucket/path?region=eu-west-1

When you use this URI as a sink, HERE Anonymizer Self-Hosted writes anonymized data to the configured S3 location.

Each sink message is written as a separate S3 object. The object key is built from the configured path and the sink message ID.
The sink does not convert message bodies to Parquet.

Authentication

This connector supports two authentication methods:

• Default authentication

  This method is used when credentials are not provided in the connector URI, for example: s3+parquet+batch+files://bucket/path?region=...
  In that case, the default AWS SDK credentials provider (reading environment variables, system properties, ~/.aws/credentials) is used.

• Explicit (static) credentials provider

  This method is used when the credentials are provided explicitly in the connector URI.
  For source usage: s3+parquet+batch+files://aws-access-id:aws-secret-key@bucket/path?region=...
  For sink usage: s3+batch+files://aws-access-id:aws-secret-key@bucket/path?region=...

Configuration

Use the following parameters to configure the connector.

The region parameter

Mandatory parameter. Defines the AWS region.

The endpoint parameter

Optional parameter. Enables an alternatively hosted S3 API, such as a LocalStack deployment.

The path_resolver_recursive parameter

Optional parameter. Defines whether S3 keys are traversed recursively by using / as a directory-like delimiter.

This parameter applies to source usage only.

Default value: true

The path_resolver_filter parameter

Optional parameter. Defines the PathMatcher pattern that filters the S3 object list relative to the bucket and path in the URI.

This parameter applies to source usage only.

Default value: glob:*

For Parquet input, limit scanned objects to Parquet files when possible:
s3+parquet+batch+files://bucket/path?region=eu-west-1&path_resolver_filter=glob:*.parquet