How to use Data Archiving Library metrics

How to use Data Archiving Library metrics

The Data Archiving Library provides a following set of metrics to monitor the archiving process:

Add user-defined metric

While the Data Archiving Library provides a comprehensive set of the archiving process metrics, the user has the option to define their own metrics using the standard Flink APIs. To define a new custom metric:

• Add your custom metric as an instance variable in your UDF implementation

public class MyExampleImplementation implements UDF {
private Counter customMetric;
...
}

• Override the open(Configuration parameters, RuntimeContext runtimeContext) API and register the custom metric which you added in the previous step with the MetricGroup of the provided RuntimeContext. For example:

...
@Override
public void open(Configuration parameters, RuntimeContext runtimeContext) {
    customMetric = runtimeContext.getMetricGroup().counter("myCustomMetric");
}

• Record the metric in your UDF implementation:

customMetric.inc();

• The following dependencies should be added with provided scope. Note that this example is for maven build tool.

<dependency>
   <groupId>org.apache.flink</groupId>
   <artifactId>flink-core</artifactId>
   <scope>provided</scope>
</dependency>
<dependency>
   <groupId>org.apache.flink</groupId>
   <artifactId>flink-metrics-core</artifactId>
   <scope>provided</scope>
</dependency>

Refer to com/here/platform/data/archive/example/AvroSimpleKeyExample.java in the HERE Data SDK for Java & Scala Data Archive Avro Example for an example implementation.

Monitoring metrics

Metrics registered with the Flink runtime context will be available in Grafana prefixed with flink_taskmanager_job_task_operator_. For example:

flink_taskmanager_job_task_operator_messageCounter
flink_taskmanager_job_task_operator_myCustomMetric

Refer to the Flink documention to learn more about the scope and naming of metrics.