How to delete versioned layer data

The Data Client Library provides the class LayerUpdater to perform update operations on versioned layers.

The LayerUpdater has 3 methods:

updateLayer(catalogHrn, layerId) defines the catalog and layer which should be updated.
option(key, value) is not applicable for versioned layers but used by e.g. index layers.
delete(queryString) performs the delete operation according the query string. The query string is in RSQL format. The delete function call is blocking/synchronous. It returns when the delete operation finished.

Project dependencies

If you want to create an application that uses the HERE platform Spark Connector to delete data from versioned layer, add the required dependencies to your project as described in chapter Dependencies for Spark Connector.

Examples

The following snippet demonstrates how to delete data from a versioned layer of a catalog.

import com.here.platform.data.client.spark.LayerDataFrameReader.SparkSessionExt
import org.apache.spark.sql.SparkSession
val df = sparkSession
  .updateLayer(catalogHrn, layerId)
  .delete(s"mt_partition=in=($partitionId, $anotherPartitionId)")

val deleteResult = df.select("result").first().getString(0)
val deletedCount = df.select("count").first().getInt(0)
val deletionMessage = df.select("message").first().getString(0)

import static org.apache.spark.sql.functions.*;

import com.here.hrn.HRN;
import com.here.platform.data.client.spark.javadsl.JavaLayerUpdater;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.IntegerType;
Dataset<Row> dataFrame =
    JavaLayerUpdater.create(sparkSession)
        .updateLayer(catalogHrn, layerId)
        .delete(String.format("mt_partition=in=(%s,%s)", partitionId, anotherPartitionId));

String deleteResult = dataFrame.select("result").first().getString(0);
int deletedCount = dataFrame.select("count").first().getInt(0);
String deletionMessage = dataFrame.select("message").first().getString(0);

Note

The payload of a versioned layer cannot be deleted because the associated blob is still used by previous versions. Therefore, the delete function of a versioned layer removes the partition from the metadata only that it does not appear in the latest version of the layer anymore.

For information on RSQL, see RSQL.

Updated 22 days ago