How to filter input partitions

Sometimes, you may have tasks that only need a subset of the input layer
partitions to produce their output layers. In such cases, it is usually
desirable to exclude unnecessary partition keys as early as possible in
the execution process to avoid wasting CPU, memory, and network bandwidth.

While it is possible to manually filter the input keys in the compiler's
front-end implementation for most compilation patterns, for some patterns,
such as the RefTreeCompiler, it can be
difficult to know if a key should be filtered, or not, since we typically
do not know if a subject or a referenced partition is being processed.

For these cases, the library provides a way to configure a partition filter
to select partition keys and metadata from the list of partitions to process.
This configuration works as follows:

• For the RefTreeCompiler, only the subject
  partitions are filtered. This is something to remember, particularly if an
  input layer is itself the product of an upstream compiler in a
  multi-compiler driver task with filtering applied. If a compiler down the
  chain references neighbor partition names from a layer produced by an
  upstream compiler, make sure that the filtering is permissive enough to
  output all partitions to be referenced by any following compilation in the
  driver task. You can use the byId executor config to
  configure a different filter for a compiler.
• The here.platform.data-processing.executors.partitionKeyFilters
  configuration is part of fingerprints, which means that changing this
  configuration triggers a non-incremental run for the compilation to remain
  deterministic.
• Partition key filters can also be used with the DeltaSet interface. Partition key filters
  defined in here.platform.data-processing.deltasets.partitionKeyFilters are used in all
  query and readBack DeltaSet transformations to filter the input partitions.

Configure filters

Partition key filtering can be configured in the application.conf.
For example, this filter can configure a task to only process partitions
contained within a given latitude/longitude bounding box:

// Note: use here.platform.data-processing.deltasets.partitionKeyFilters to filter deltasets.
here.platform.data-processing.executors.partitionKeyFilters = [
  {
    className = "BoundingBoxFilter"
    param.boundingBox { north = 24.8, south = 24.68, east = 121.8, west = 121.7 }
  }
]

The root of the property is a list, multiple filters specified at this level
are combined as their union (OR logic). If there is a need to combine
them with AND logic, they can be put under a single AndFilter at the root.

This is the list of built-in filters:

• BoundingBoxFilter
• AllowListFilter
• AndFilter
• OrFilter
• NotFilter

A custom filter can also be applied from applications by extending the
PartitionKeyFilter. These filters return either true or false from
shouldProcess. Boolean logic is applied to combine them up to the root
filter, such as:

• Or of two bounding boxes is a union
• And of two bounding boxes is an intersection
• Notof two bounding boxes means that only partitions outside of the
  underlying bounding box are considered

Filters can only be applied on partition key parameters, for example, catalog
and layer IDs, and partition name.

Note

For performance reasons, filters do not filter based on partition
metadata or payloads.

Override filters for specific compilers

If one of the compilers needs a different partition filter applied, you can
use the byId mechanism to configure a different set of filter for the
executor that wraps it. Overriding these filters means replacing the whole
default set of filters, as they are not combined.

Example:

// Application-wide default filter.
here.platform.data-processing.executors.partitionKeyFilters = [
  {
    className = "BoundingBoxFilter"
    param.boundingBox { north = 24.8, south = 24.68, east = 121.8, west = 121.7 }
  }
]

// Filters can be overridden for specific executors/compilers using their Executor.Id.
here.platform.data-processing.executors.byId {
  // Apply a larger bounding box filter to the input layers of this compiler to
  // include neighbor partitions.
  intermediate-compiler.partitionKeyFilters = [
    {
      className = "BoundingBoxFilter"
      param.boundingBox { north = 24.9, south = 24.58, east = 121.9, west = 121.6 }
    }
  ]
  // This compiler isn't using filtering.
  another-compiler.partitionKeyFilters = []
}

Apply a filter to specific layers only

The AllowListFilter can be used to filter based on a fixed list of partition
names. When combined with Boolean operation filters, AllowListFilter can also
be used to apply some filters to specific layers only.

For example, to apply a bounding box only to inLayer of inCatalog, you can
configure your application as follows:

// Note: use here.platform.data-processing.deltasets.partitionKeyFilters to filter deltasets.
// Match (process) a partition key if:
here.platform.data-processing.executors.partitionKeyFilters = [
  // NOT layer in inCatalog:inLayer
  {
    className = "NotFilter"
    param.operand = {
      className = "AllowListFilter"
      param.catalogsAndLayers = {"inCatalog": ["inLayer"]}
    }
  },
  // OR (layer in inCatalog:inLayer AND partition intersects boundingBox)
  {
    className = "AndFilter"
    param.operands = [
      {
        className = "AllowListFilter"
        param.catalogsAndLayers = {"inCatalog": ["inLayer"]}
      }, {
        className = "BoundingBoxFilter"
        param.boundingBox { north = 2.8, south = 2.68, east = 121.8, west = 121.7 }
      }
    ]
  }
]

For additional information about configuring partition key filters see
Configure The Library.