Ingestion Metrics dashboard

Ingestion Metrics is a default dashboard available to you in Grafana that shows the following metrics.

For the list of Ingestion metrics, labels and their possible values refer to the Common Metrics and Labels for Ingestion section at the end of this chapter.

Messages Received By the HERE Platform Ingestion Endpoints

METRIC NAME	DETAILS	DESCRIPTION
Total number of all messages received from realm	Metric `incoming_requests_total`, with labels `data_type="single"` and `endpoint_name="ingest"`	Total number of all messages received (by HERE platform ingestion endpoints) from the organization / realm. Does not include messages having failed authentication or authorization or those not received due to network/other issues.
Total number of messages received	Metric `incoming_requests_total`, with labels `data_type="single"` `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Total number of messages received (by HERE platform ingestion endpoints) by Catalog by Layer. Does not include messages having failed authentication or authorization or those not received due to network/other issues.
Total size of all messages received from realm	Metric `raw_size_total`, with labels `data_type="single"` and `endpoint_name="ingest"`	Total size of all messages received from the organization / realm
Total size of messages received	Metric `raw_size_total`, with labels `data_type="single"` `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Total size of messages received by Catalog by Layer
Average message size received	Metric `raw_size_stats_mean`, with labels `data_type="single"` `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Average size of messages received by Catalog by Layer
Minimum message size received	Metric `raw_size_stats_min`, with labels `data_type="single"` `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Minimum size of messages received by Catalog by Layer
Maximum message size received	Metric `raw_size_stats_max`, with labels `data_type="single"` `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Maximum size of messages received by Catalog by Layer

Messages Successfully Ingested By the HERE Platform

METRIC NAME	DETAILS	DESCRIPTION
Total number of all messages ingested from realm	Metric `successful_requests_total`, with labels `data_type="single"` and `endpoint_name="ingest"`	Total number of all messages successfully ingested from the organization / realm.
Total number of messages ingested	Metric `successful_requests_total`, with labels `data_type="single"`, `endpoint_name="ingest"`, `aid`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Total number of messages successfully ingested by Catalog by Layer
Total size of all messages ingested	Metric `raw_size_ingested_total`, with labels `data_type="single"` and `endpoint_name="ingest"`	Total size of all messages ingested from the organization / realm
Total size of messages ingested	Metric `raw_size_ingested_total`, with labels `data_type="single"` `endpoint_name="ingest"`, `aid`,`catalog` and `layer`, grouped by `catalog` and `layer`.	Total size of messages ingested by Catalog by Layer
Average message size ingested	Metric `raw_size_stats_ingested_mean`, with labels `data_type="single"`, `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Average size of messages ingested by Catalog by Layer
Minimum message size ingested	Metric `raw_size_stats_ingested_min`, with labels `data_type="single"`, `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Minimum size of messages ingested by Catalog by Layer
Maximum message size ingested	Metric `raw_size_stats_ingested_max`, with labels `data_type="single"`, `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Maximum size of messages ingested by Catalog by Layer
Total number of message ingestion failures	Metric `failed_requests_total`, with labels `data_type="single"`, `endpoint_name="ingest"`, `aid`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Total number of message ingestion failures (messages received but not ingested) by Catalog by Layer (for example, 400’s and 500’s, but not failures due to authorization failures or nonexistent catalog destinations).

Message Rate / Throughput of Ingested Messages

METRIC NAME	DETAILS	DESCRIPTION
Number of bytes per second (Mean and Max)	Metric `raw_size_ingested_total`, with labels `data_type="single"`, `endpoint_name="ingest"`, `aid`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Number of bytes ingested per second by Catalog by Layer
Number of messages per second (Mean and Max)	Metric `successful_requests_total`, with labels `data_type="single"`, `endpoint_name="ingest"`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Number of messages ingested per second by Catalog by Layer

SDII Specific Data Ingestion Metrics by the HERE Platform SDII Ingestion Endpoint

METRIC NAME	DETAILS	DESCRIPTION
Number of messages received by the platform SDIIMessageList endpoint	Metric `incoming_requests_total`, with labels `data_type="multi"`, `endpoint_name="ingest"`, `aid`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Number of SDII multi-messages received (by `SDIIMessage` endpoint) by Catalog by Layer
Number of messages ingested by the platform SDIIMessageList endpoint	Metric `successful_requests_total`, with labels `data_type="multi"`, `endpoint_name="ingest"`, `aid`, `catalog` and `layer`, grouped by `catalog` and `layer`.	Number of SDII multi-messages ingested (by `SDIIMessage` endpoint) by Catalog by Layer
Number of single messages contained within MessageLists sent to the platform SDIIMessageList endpoint	Metric `num_of_messages_total`, with labels `data_type="multi"`, `endpoint_name="ingest"`, `aid`, `catalog` and `layer`, grouped by `catalog` and `layer`.	SDII multi-messages or "MessageList" include one or more single SDII messages. This number is the sum of single messages included in those multi-messages by Catalog by Layer.

SDII Specific Data Ingestion Metrics by Sensor Data Ingestion Platform Ingestion Endpoints (legacy endpoints)

METRIC NAME	DETAILS	DESCRIPTION
Number of messages received by the `sensorData` endpoint	Metric `incoming_requests_total`, with labels `data_type="single"`, `endpoint_name="sdip.*"` and `aid`, grouped by `catalog`, `layer` and `aid`.	Number of SDII messages received (by `sensorData` endpoint) by AppID.
Number of messages ingested by the `sensorData` endpoint	Metric `successful_requests_total`, with labels `data_type="single"`, `endpoint_name="sdip.*"` and `aid`, grouped by `catalog`, `layer` and `aid`.	Number of SDII messages ingested (by `sensorData` endpoint) by AppID.
Number of message ingestion failures by the `sensorData` endpoint	Metric `failed_requests_total`, with labels `data_type="single"`, `endpoint_name="sdip.*"` and `aid`, grouped by `catalog`, `layer` and `aid`.	Number of SDII message ingestion failures (not ingested) by AppID.
Number of messages received by the `sensorDataMulti` endpoint	Metric `incoming_requests_total`, with labels `data_type="multi"`, `endpoint_name="sdip.*"` and `aid`, grouped by `catalog`, `layer` and `aid`.	Number of SDII messages received (by `sensorDataMulti` endpoint) by AppID.
Number of messages ingested by the `sensorDataMulti` endpoint	Metric `successful_requests_total`, with labels `data_type="single"`, `endpoint_name="sdip.*"` and `aid`, grouped by `catalog`, `layer` and `aid`.	Number of SDII messages ingested (by `sensorDataMulti` endpoint) by AppID.
Number of single messages contained within MessageLists by `sensorDataMulti` endpoint	Metric `num_of_messages_total`, with labels `data_type="single"`, `endpoint_name="sdip.*"` and `aid`, grouped by `catalog`, `layer` and `aid`.	SDII multi-messages or "MessageList" include one or more single SDII messages. This number is the sum of single messages included in those multi-messages by AppID.
Number of SDII message ingestion failures by the `sensorDataMulti` endpoint	Metric `failed_requests_total`, with labels `data_type="single"`, `endpoint_name="sdip.*"` and `aid`, grouped by `catalog`, `layer` and `aid`.	Number of SDII message ingestion failures (not ingested) by AppID.

Common Metrics and Labels for Ingestion

Below the most common metrics for Ingestion.

incoming_requests_total
failed_requests_total
num_of_messages_total
successful_requests_total
raw_size_total
raw_size_stats_min
raw_size_stats_max
raw_size_stats_mean
raw_size_ingested_total
raw_size_stats_ingested_min
raw_size_stats_ingested_max
raw_size_stats_ingested_mean

All the Ingestion metrics can be filtered out with these labels:

LABEL NAME	DESCRIPTION
`aid`	to filter out for a particular application id.
`catalog`	to filter out for a particular catalog id.
`data_type`	to filter for incoming request with a single or multiple messages in the body, possible values are `single` or `multi`.
`endpoint_name`	to filter out for a particular ingestion endpoint, possible values are `ingestion` or `sdip.*` (legacy endpoint). For the standard Ingestion service use the `ingest` value.
`layer`	to filter out for the layer id.