GuidesChangelogData Inspector Library API Reference
Guides

How to create catalogs

How to create catalogs

To manage catalogs, add the data-client module as a dependency to your project.

Configure catalogs

Catalog configurations include the properties listed in the following table. Mandatory properties are in bold.

Note

Catalog IDs
Catalog IDs are publicly visible since they are part of the catalog HRN. Do not include private or company confidential information when you specify a catalog ID. Catalog names are private by default, which means you can add your private or confidential information in this field.

PropertyDescription
idIDs are part of the HERE Resource Name (HRN) for a catalog.
nameA short-form name for the catalog.
summaryA one sentence summary describing the catalog. This summary is visible when you browse a list of catalogs on the platform portal.
descriptionA detailed description of the catalog and its content. This information appears on the 'Overview' tab when you open a catalog in the platform portal.
layersThe layers in the catalog.
tagsSome keywords that help when searching for the catalog in the platform portal.
notificationsEnabledIf enabled, a notification topic for the catalog is created. This topic allows an application to receive a notification when the catalog is updated.
automaticVersionDeletionIf set, enables automatic deletion of catalog versions to manage storage costs and enable lifecycle management of versioned layers.
areaSchemeThe HERE Resource Name (HRN) of the area scheme used across the catalog. If the value is not set than no area scheme is applied to the catalog.
subsetRestrictionsIndicates if the subsetRestrictions for a catalog is enabled or not. subsetRestrictions can only be enabled for catalogs that are defined within projects. It requires area scheme being set. If catalog subsetRestrictions are disabled, supported layer's subsetRestrictions will be disabled by default.

Layer configurations include the properties listed in the table below.

PropertyDescription
idIDs must be unique for all layers within a catalog.
nameA short-form name for the layer.
summaryA one sentence summary describing the layer. This summary is visible when you browse a list of layers on the platform portal.
descriptionA detailed description of the layer and its contents. This information appears on the 'Overview' tab when you open a layer in the platform portal.
layerTypeThe type of data provided by the layer: versioned, volatile, stream, index, interactivemap or objectstore.
volumeTypeProperties that describe the volume where the data is physically stored: durable or volatile.
partitioningThe name of the partitioning scheme for this layer: generic or heretile. This property is not supported by the objectstore layer type.
contentTypeThe MIME type of the data stored in the layer. This property is not supported by the object store layer type.
contentEncodingIndicator if the data is compressed. Currently, gzip is supported.
coverageA set of admin area strings ( ISO 3166 two-letter codes for countries and regions optionally followed by up to three characters codes for subdivisions, such as 'DE', 'PL' or 'CN-HK'), which define the geographic area that this layer covers.
tagsA list of keywords that help to find the layer when searching in the platform portal.
billing tagsA list of billing tags a free-form tag, which is used for grouping billing records together. If supplied, it must be between 4 - 16 characters, contain only alphanumeric ASCII characters [A-Za-z0-9].
schemaThe HRN of the data schema for this layer.
digestThe digest algorithm used by the data publisher to generate a checksum for each partition in the layer. Data consumers can use it to verify the integrity of the data they retrieve from the layer. Currently, MD5, SHA-1 and SHA-256 are supported.
crcThe CRC algorithm used to calculate the checksum for the partitions in this layer. Currently, CRC-32C is supported.

Versioned layer configuration

Versioned layers have the following additional configuration fields.

PropertyDescription
secureHandlesA boolean flag which indicates if the layer metadata and blob are in mode of secure data handles support. It requires catalog area scheme to be set and supports only heretile partitioning of the layer.
subsetRestrictionsA boolean flag which indicates if the subsetRestrictions for a layer is enabled or not. subsetRestrictions can only be enabled for layers of catalogs that are defined within projects. It requires area scheme of the catalog being set. Supported layer subsetRestrictions are enabled by default when subsetRestrictions is enabled for the catalog. Layer subsetRestrictions requires catalog subsetRestrictions to be enabled. If catalog subsetRestrictions are disabled, supported layer's subsetRestrictions will be disabled by default. Supported only for layers with heretile partitioning.

Volatile layer configuration

Volatile layers have the following additional configuration fields.

PropertyDescription
ttlThe expiry time in milliseconds for data in this layer. Data is automatically removed after the specified time limit has elapsed. For volatile layers the TTL value must be between 60000 (1 minute) and 604800000 (7 days).
dataRedundancyThe data redundancy for volatile layers. Possible enum values are single-instance or multi-instance. Single instance layers do not have data redundancy, whereas multi instance layers do.
storageCapacityMbThe storage capacity of the layer specified in megabytes, with an increment step of 100 (eg. 100, 500, 600, 2000). For volatile layers, the storage capacity must be between 100 (100MB) and 21000 (21GB).
secureHandlesA boolean flag which indicates if the layer metadata and blob are in mode of secure data handles support. It requires catalog area scheme to be set and supports only heretile partitioning of the layer.
subsetRestrictionsA boolean flag which indicates if the subsetRestrictions for a layer is enabled or not. subsetRestrictions can only be enabled for layers of catalogs that are defined within projects. It requires area scheme of the catalog being set. Supported layer subsetRestrictions are enabled by default when subsetRestrictions is enabled for the catalog. Layer subsetRestrictions requires catalog subsetRestrictions to be enabled. If catalog subsetRestrictions are disabled, supported layer's subsetRestrictions will be disabled by default. Supported only for layers with heretile partitioning.

Index layer configuration

Index layers have the following additional configuration fields.

PropertyDescription
indexDefinitionsThe collection of index key definitions to index published partitions.
ttlTime to Live of indexed data in index layer. If not specified it defaults to 7 days.

Each index key definition contains the following configuration fields:

PropertyDescription
nameThe name of index key. Max name length is 64 characters. The name must begin with a Unicode letter. Subsequent characters can be letters, underscores (_), digits (0-9)
indexTypeThe type of index key. Supported types: Int, Bool, String, TimeWindow, and HereTile.
durationDuration attribute of the TimeWindow type must be in the range 600000 (10 minutes) .. 86400000 (24h in millis)
zoomLevelzoomLevel attribute for HereTile type must be in 0 .. 14 range

Supported index key types:

PropertyDescription
IntA signed integer, up to 64 bits.
BoolA Boolean value.
StringA string of Unicode characters with the maximum length of 40.
TimeWindowRepresents the time slice (NOT just point in time) and it denotes the finest time granularity at which the data will be indexed and later queried. Time type has an attribute duration which represents the time slice length and similarly to the here-tile is NOT mutable. Both here-time value and here-time duration attribute is expressed in milliseconds (time value is milliseconds since Epoch).
HereTileRepresents the tile id in the HERE tile map tiling scheme. HereTile type has an attribute zoomLevel which represents the size of the tile and it's NOT mutable.

The TimeWindow index key is required. Index layers can have at most a single TimeWindow and a single HereTile index key. This means that Index layers can have exactly one TimeWindow index key and only one optional HereTile index key.

Below are the possible TTL values:

Enum ValueTTL
SevenDays7 days
FifteenDays15 days
OneMonth30 days
TwoMonths60 days
ThreeMonths90 days
FourMonths120 days
SixMonths180 days
OneYear365 days
OneAndHalfYears548 days
TwoYears730 days
Unlimitedunlimited

Stream layer configuration

Stream layers have the following additional configuration fields.

PropertyDescription
dataInThroughputKbpsMaximum throughput for incoming data expressed in kilobytes per second. Throttling occurs when the inbound rate exceeds the maximum inbound throughput. The default is 1000 KBps. The minimum is 100 KBps, the maximum is 32800 KBps. Can only be multiple of 100 KBps.
dataOutThroughputKbpsMaximum throughput for outgoing data expressed in kilobytes per second. Throttling occurs when the total outbound rate to all consumers exceeds the maximum outbound throughput. The default is 4000 KBps. The minimum is 100 KBps, the maximum is 65500 KBps. Can only be multiple of 100 KBps.
parallelizationThe number of Kafka partitions for Stream Layer. The default is 4, the maximum is 32.
ttlThe expiry time in milliseconds for data in this layer. Data is automatically removed after the specified time limit has elapsed. For stream layers, the TTL value must be between 600000 (10 minutes) and 259200000 (72 hours).

Note

General recommendation for stream layer properties
(dataInThroughputKbps and dataOutThroughputKbps and parallelization) is to provide all of them in the request.

To preserve backward compatibility, there are few corner cases:

  1. User should provide either dataInThroughputKbps and dataOutThroughputKbps or parallelization.
  2. If all 3 fields are provided, then we use them as is.
  3. If all stream properties are missing, then default values dataInThroughputKbps = 1000, dataOutThroughputKbps = 4000, parallelization = 4 will be set

Interactive Map layer configuration

Interactive Map layers have the following additional configuration fields.

PropertyDescription
Searchable PropertiesInteractive map layers can be configured with 'Searchable Properties'. This is a list of strings containing the properties you want to index for faster searching. Searchable properties can be updated at any time. A maximum of eight searchable properties can be added.

Automatic version deletion configuration

Warning

Data loss
Deleting Catalog versions will delete partition metadata AND data associated with those versions, impacting all versioned layers in the catalog. The partition metadata and data will be irrecoverable. Any partition metadata and data from deleted versions that is still used in current, non-deleted versions will not be deleted so that non-deleted versions remain functional.

Supported automaticVersionDeletion types:

PropertyDescription
numberOfVersionsToKeepIf set and once the max number of versions is reached, every time a new version is created, the oldest version will be deleted.

Note

For instructions on how to disable automatic version deletion, see Disable an Automatic Version Deletion

Create catalogs

To configure and create catalogs, add the following:

val catalogId = "creating-catalog-test-" + someIdentifierSuffix
val catalogConfig =
  WritableCatalogConfiguration(
    id = catalogId,
    name = catalogId,
    summary = "This is a catalog summary.",
    description = "This is what the catalog is for.",
    tags = Set("tag1", "tag2", "tag3"),
    automaticVersionDeletion =
      Some(AutomaticVersionDeletion.builder.withNumberOfVersionsToKeep(10L).build),
    areaScheme = Some(HRN.fromString("hrn:here:areascheme::olp-here:default-countries")),
    subsetRestrictions = Some(true),
    layers = Seq(
      WritableLayer(
        id = "layer-0",
        name = "INFO",
        summary = "This is a layer summary.",
        description = "This is a layer description.",
        layerType = VersionedLayerType(),
        partitioning = GenericPartitioning,
        volume = Volumes.Durable,
        contentType = "application/x-protobuf",
        digest = Some(DigestAlgorithm.MD5),
        crc = Some(CrcAlgorithm.CRC32C)
      ),
      WritableLayer(
        id = "layer-1",
        name = "SCHEMAS",
        summary = "This is a layer summary.",
        description = "This is a layer description.",
        layerType = VersionedLayerType(),
        partitioning = GenericPartitioning,
        volume = Volumes.Durable,
        contentType = "application/x-protobuf"
      ),
      WritableLayer(
        id = "layer-2",
        name = "VOLATILEDATA",
        summary = "This is a layer summary.",
        description = "This is a layer description.",
        layerType =
          VolatileLayerType(secureHandles = Some(true), subsetRestrictions = Some(true)),
        partitioning = HereTilePartitioning(tileLevels = 1 :: Nil),
        volume = Volumes.Durable,
        contentType = "application/octet-stream",
        contentEncoding = Some(ContentEncoding.gzip),
        coverage = Some(Coverage(Set[String]("DE", "US")))
      ),
      WritableLayer(
        id = "layer-3",
        name = "INDEX",
        summary = "This is a layer summary.",
        description = "This is a layer description.",
        layerType = IndexLayerType(
          indexDefinitions = Seq(
            IndexDefinition("someIntKey", IndexType.Int),
            IndexDefinition("someStringKey", IndexType.String),
            IndexDefinition("someTimeWindowKey",
                            IndexType.TimeWindow,
                            duration = Some(3600000)),
            IndexDefinition("someHereTileKey", IndexType.HereTile, zoomLevel = Some(8))
          ),
          ttl = Ttl.OneMonth
        ),
        partitioning = NoPartitioning,
        volume = Volumes.Durable,
        contentType = "application/octet-stream",
        digest = Some(DigestAlgorithm.SHA256),
        crc = Some(CrcAlgorithm.CRC32C)
      ),
      WritableLayer(
        id = "layer-4",
        name = "OBJECTSTORE",
        summary = "This is layer summary",
        description = "This is layer description",
        layerType = ObjectStoreLayerType(),
        partitioning = NoPartitioning,
        volume = Volumes.Durable
      ),
      WritableLayer(
        id = "layer-5",
        name = "INTERACTIVEMAP",
        summary = "This is layer summary",
        description = "This is layer description",
        layerType = InteractiveMapLayerType(
          interactiveMapProperties = Some(
            InteractiveMapProperties(searchableProperties = Seq("abc"),
                                     versionsToKeep = Some(123L)))
        ),
        partitioning = NoPartitioning,
        volume = Volumes.Durable
      )
    )
  )

val adminApi = DataClient().adminApi()
adminApi.createCatalog(catalogConfig).flatMap { hrn =>
  log.info(s"Created new catalog `$hrn`")
  processCreatedCatalog(hrn)
}
String catalogId = "creating-catalog-test-" + someIdentifierSuffix;
WritableCatalogConfiguration catalogConfig =
    new WritableCatalogConfiguration.Builder()
        .withId(catalogId)
        .withName(catalogId)
        .withSummary("This is a catalog summary.")
        .withDescription("This is what the catalog is for.")
        .withTags(new HashSet<>(Arrays.asList("tag1", "tag2", "tag3")))
        .withAutomaticVersionDeletion(
            AutomaticVersionDeletion.builder().withNumberOfVersionsToKeep(10L).build())
        .withLayers(
            Arrays.asList(
                new WritableLayer.Builder()
                    .withId("layer-0")
                    .withName("INFO")
                    .withSummary("This is a layer summary.")
                    .withDescription("This is a layer description.")
                    .withLayerType(VersionedLayerType.defaultInstance())
                    .withPartitioning(Partitioning.Generic())
                    .withVolume(Volumes.Durable())
                    .withContentType("application/x-protobuf")
                    .withDigest(DigestAlgorithm.SHA256())
                    .withCrc(CrcAlgorithm.CRC32C()),
                new WritableLayer.Builder()
                    .withId("layer-1")
                    .withName("SCHEMAS")
                    .withSummary("This is a layer summary.")
                    .withDescription("This is a layer description.")
                    .withLayerType(VersionedLayerType.defaultInstance())
                    .withPartitioning(Partitioning.Generic())
                    .withVolume(Volumes.Durable())
                    .withContentType("application/x-protobuf"),
                new WritableLayer.Builder()
                    .withId("layer-2")
                    .withName("INDEX")
                    .withSummary("This is a layer summary.")
                    .withDescription("This is a layer description.")
                    .withLayerType(
                        new IndexLayerType.Builder()
                            .addIndexDefinition(
                                new IndexDefinition.Builder()
                                    .withName("someIntKey")
                                    .withIndexType(IndexType.Int)
                                    .build())
                            .addIndexDefinition(
                                new IndexDefinition.Builder()
                                    .withName("someStringKey")
                                    .withIndexType(IndexType.String)
                                    .build())
                            .addIndexDefinition(
                                new IndexDefinition.Builder()
                                    .withName("someTimeWindowKey")
                                    .withIndexType(IndexType.TimeWindow)
                                    .withDuration(3600000L)
                                    .build())
                            .addIndexDefinition(
                                new IndexDefinition.Builder()
                                    .withName("someHereTileKey")
                                    .withIndexType(IndexType.HereTile)
                                    .withZoomLevel(8)
                                    .build())
                            .withTtl(Ttl.OneMonth)
                            .build())
                    .withPartitioning(Partitioning.NoPartition())
                    .withVolume(Volumes.Durable())
                    .withContentType("application/x-protobuf"),
                new WritableLayer.Builder()
                    .withId("layer-3")
                    .withName("OBJECTSTORE")
                    .withSummary("This is layer summary")
                    .withDescription("This is layer description")
                    .withLayerType(ObjectStoreLayerType.defaultInstance())
                    .withPartitioning(Partitioning.NoPartition())
                    .withVolume(Volumes.Durable()),
                new WritableLayer.Builder()
                    .withId("layer-4")
                    .withName("INTERACTIVEMAP")
                    .withSummary("This is layer summary")
                    .withDescription("This is layer description")
                    .withLayerType(
                        new InteractiveMapLayerType.Builder()
                            .withInteractiveMapProperties(
                                new InteractiveMapProperties.Builder()
                                    .withSearchableProperties(
                                        Arrays.asList("some-property1", "some-property-2"))
                                    .build())
                            .build())
                    .withPartitioning(Partitioning.NoPartition())
                    .withVolume(Volumes.Durable())
                    .withContentType("application/geo+json")))
        .build();

AdminApi adminApi = DataClient.get(myActorSystem).adminApi();
adminApi
    .createCatalog(catalogConfig)
    .thenApply(
        hrn -> {
          log.info("Created new catalog `" + hrn + "`");
          return processCreatedCatalog(hrn);
        });

Note

It may take several minutes for a new catalog to appear in the list of catalogs returned by the Data Client Library, CLI, REST API, and the list displayed on the HERE platform portal.