Data, Catalog, and Layer Metrics dashboard

Data, Catalog, and Layer Metrics is a default dashboard available to you in Grafana that shows the following metrics. The Technical Name column shows the metric name to use if you want to create a custom report. If the metric is calculated using a filter, then both the technical name and the filter are shown.

Data Metrics

METRIC	DETAILS	DESCRIPTION
Total Number of Catalogs	`catalog_total`	The number of existing catalogs created over the selected time period
Total Number of Layers	`layer_total`	The number of existing layers (all layer types) created across all catalogs over the selected time period
Total Number of Versioned Layers	`layer_total` with `layer_type="versioned"`	The number of versioned layers created across all catalogs over the selected time period
Total Number of the Object Store Layers	`layer_total` with `layer_type="objectstore"`	The number of the object store layers created across all the catalogs over the selected time period
Total Number of Volatile Layers	`layer_total` with `layer_type="volatile"`	The number of volatile layers created across all catalogs over the selected time period
Total Number of Stream Layers	`layer_total` with `layer_type="stream"`	The number of stream layers created across all catalogs over the selected time period
Total Number of Index Layers	`layer_total` with `layer_type="index"`	The number of index layers created across all catalogs over the selected time period
Versioned Layers - Total Storage Volume (Usage)	`layer_used_bytes` with `layer_type="versioned"`	Sum of data volume across all versioned layers (all catalogs)
Object store Layers - Total Storage Volume (Usage)	`layer_used_bytes` with `layer_type="objectstore"`	Sum of data volume across all the object store layers (all catalogs)
Volatile Layers - Total Capacity	`layer_allocated_bytes` with `layer_type="volatile"`	Sum of reserved capacity across all volatile layers (all catalogs)
Volatile Layers - Used Capacity	`layer_used_bytes` with `layer_type="volatile"`	Sum of used capacity across all volatile layers (all catalogs)
Stream Layers - Total Throughput Capacity	`in_throughput_kbps` with `layer_type="stream" + out_throughput_kbps` with `layer_type="stream"`	Sum of throughput capacity across all stream layers (all Catalogs)
Index Layers - Total Storage Volume (Usage)	`layer_used_index_blobstore_bytes` with `layer_type="index"`	Sum of data volume across all Index layers (all catalogs)

Catalog Metrics

METRIC	DETAILS	DESCRIPTION
Layer Count Per Catalog	`layer_total_by_catalog`	Total Number of Layers Per Catalog
Versioned Layer Storage Volume Per Catalog	`layer_used_bytes` with `layer_type="versioned"` and `catalog=~"$catalog"`	Total Data Volume of all Versioned Layers per Catalog
Object Store Layer Storage Volume Per Catalog	`layer_used_bytes` with `layer_type="objectstore"` and `catalog=~"$catalog"`	Total Data Volume of all Object Store Layers per Catalog
Volatile Layer Unique Capacity Per Catalog	`layer_unique_used_bytes` with `layer_type="volatile"` and `catalog=~"$catalog"`	Total Unique Capacity (Amount of Stored Data Without Replication) of all volatile layers per catalog
Index Layer Data Storage Volume Per Catalog	`layer_used_index_blobstore_bytes` with `layer_type="index"` and `catalog=~"$catalog"`	Total data volume of all index layers per catalog

Volatile Layer Metrics

METRIC	DETAILS	DESCRIPTION
Capacity Allocated for Volatile Storage	`layer_allocated_bytes`	Total capacity configured
Capacity Used for Volatile Storage	`layer_used_bytes`	Total capacity used
Unique Capacity Used for Volatile Storage	`layer_unique_used_bytes`	Amount of stored data without replication
Data Redundancy for Volatile Storage	`layer_data_redundancy`	Storage redundancy set per volatile layer configuration.

Stream Layer Metrics

METRIC	DETAILS	DESCRIPTION
TTL per Stream Layer	`ttl_min`	Time-to-Live (TTL) - User configuration setting per Stream Layer
Throughput per Stream Layer	`in_throughput_kbps` and `out_throughput_kbps`	IN/OUT throughput (Kilobytes/second) per stream layer
Write Byte Rate Per Layer (MB/s)	`kafka_server_produce_byte_rate`	Byte rate per second for all messages written to a layer
Read Byte Rate Per Layer (MB/s)	`kafka_server_fetch_byte_rate`	Byte rate per second for all messages read from a layer
Messages written Per Layer	`stream_layer_messagesInRate`	Total number of messages written per stream layer. (For Dashboard UI displays use max() of this metric)

Versioned Layer Metrics

METRIC	DETAILS	DESCRIPTION
Durable Storage Used for Versioned Layer	`layer_used_bytes`	Total capacity used for versioned layer in the catalogs

Index Layer Metrics

METRIC	DETAILS	DESCRIPTION
TTL per Index Layer	`ttl_index_min`	Time-to-Live (TTL) - User configuration settings per index layer
Metadata Storage Used for Index Layers	`index_layer_metadata_storage_bytes`	Total volume of metadata storage per index layer

Object Store Layer Metrics

METRIC	DETAILS	DESCRIPTION
Capacity Used for Object Store Layer	`layer_used_bytes`	Total capacity used for the Object Store layer in the catalogs
Metadata Storage Used for Object Store Layers	`objectstore_layer_metadata_used_bytes`	Total volume of the metadata storage per Object Store layer

Common Metrics and Labels used for Catalog and Layers

The table below reports the most common metrics and their labels for Catalogs an Layers.

METRIC	LABELS
`catalog_total`	-
`layer_total`	`layer_type` with values: `index`, `stream`, `versioned`, `volatile`
`layer_total_by_catalog`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN
`layer_used_bytes`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `clientAppId` to filter out for a particular Client App Id. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type. `project` to filter out for a particular project id.
`layer_unique_used_bytes`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type, in this label `volatile` is the only value.
`layer_allocated_bytes`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `clientAppId` to filter out for a particular Client App Id. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type, in this label `volatile` is the only value. `project` to filter out for a particular project id.
`layer_used_index_blobstore_bytes`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `clientAppId` to filter out for a particular Client App Id. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type, in this label `index` is the only value. `project` to filter out for a particular project id.
`index_layer_metadata_storage_bytes`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `clientAppId` to filter out for a particular Client App Id. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type, in this label `index` is the only value. `project` to filter out for a particular project id.
`in_throughput_kbps` and `out_throughput_kbps`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `clientAppId` to filter out for a particular Client App Id. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type, in this label `stream` is the only value. `project` to filter out for a particular project id.
`objectstore_layer_metadata_used_bytes`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `clientAppId` to filter out for a particular Client App Id. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type, in this label `objectstore` is the only value. `project` to filter out for a particular project id.
`kafka_server_fetch_byte_rate` and `kafka_server_produce_byte_rate`	`clientId` to filter out for a particular client id. `strCatalog` to filter out for a particular catalog id. `strLayer` to filter out for a particular catalog layer.
`stream_layer_messagesInRate`	`strCatalog` to filter out for a particular catalog id. `strLayer` to filter out for a particular catalog layer.
`ttl_min`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `clientAppId` to filter out for a particular Client App Id. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type, in this label `stream` is the only value. `project` to filter out for a particular project id.
`ttl_index_min`	`billingTag` to filter out for a particular billing tag. `catalog` to filter out for a particular catalog id. `catalogHrn` to filter out for a particular catalog HRN. `clientAppId` to filter out for a particular Client App Id. `layer` to filter out for a particular layer id. `layer_type` to filter out for a particular layer type, in this label `index` is the only value. `project` to filter out for a particular project id.