Data workflows

The HERE platform organizes data according to three key concepts: catalog, layer, and partition. The platform hosts a number of catalogs, each with one or more layers, which are divided into partitions.

For more information on these concepts, see the Data API.

The OLP CLI provides tools for managing data, such as:

List all catalogs to which you have access
Create a new catalog and layer
Upload data to a catalog

For more details, see catalog commands.

List all catalogs

To list the catalogs to which you have access, enter the olp catalog list command.

olp catalog list

The command lists the HRNs of the catalogs to which you have access, each in a new line.

Create a new catalog and layer

Follow the steps below to create a new catalog and layer.

Enter the olp catalog create command.

    olp catalog create first-catalog-example-id first-catalog-example \
    --summary "This catalog contains a sample layer containing data in CSV format"

    olp catalog create first-catalog-example-id first-catalog-example ^
    --summary "This catalog contains a sample layer containing data in CSV format"

To add a layer to your catalog, enter the olp catalog layer add command.

    olp catalog layer add hrn:here:data::org:first-catalog-example-id example-layer-id \
     "Example layer" --versioned --summary "This layer contains simple data in CSV format" \
     --description "A longer description of the layer contents" --content-type=text/csv

    olp catalog layer add hrn:here:data::org:first-catalog-example-id example-layer-id ^
    "Example layer" --versioned --summary "This layer contains simple data in CSV format" ^
    --description "A longer description of the layer contents" --content-type=text/csv

The command above adds a versioned layer with the ID example-layer-id and name Example layer to the hrn:here:data::org:first-catalog-example-id catalog. The content type of this layer is set to text/csv.

For Example layer, the following parameters were not explicitly used and set to their default values:

--content-encoding uncompressed The content encoding of the layer. If this parameter is specified, all data in a layer is compressed by the specified algorithm. The default value is uncompressed, which means no compression.
--partitioning generic The partitioning scheme of the layer, either heretile or generic. For heretile, you can specify tile levels separated by a comma, such as heretile:level1,level2. The platform supports tile levels from 1 to 15. For more information on HERE Tile Partitioning, see the Calculate Partition Tile IDs tutorial. The default value is generic.

To verify the catalog configuration, enter the olp catalog show command.

olp catalog show hrn:here:data::org:first-catalog-example-id --json

The command displays the following results.

{
        "summary": "This catalog contains sample layers containing simple text",
        "owner": {
            "creator": {"id": "mzLcb1rL8nskvDQpCFEF"},
            "organisation": {"id": "org"}
        },
        "billingTags": [],
        "hrn": "hrn:here:data::org:first-catalog-example-id",
        "created": "2018-03-26T12:48:10.315Z",
        "name": "first-catalog-example",
        "layers": [{
            "summary": "This layer contains simple data in CSV format",
            "volume": {
                 "volumeType": "durable"
            },
            "layerType": "versioned",
            "billingTags": [],
            "name": "Example layer",
            "description": "A longer description of the layer contents",
            "partitioningScheme": "generic",
            "partitioning": {
                 "scheme": "generic"
            },
            "id": "example-layer-id",
            "contentType": "text/csv",
            "tags": []
        }],
        "description": "",
        "notifications": {
            "enabled": false
         },
        "id": "first-catalog-example-id",
        "version": -1,
        "tags": []
    }

From the command output above, you can see that a catalog with HRN hrn:here:data::org:first-catalog-example-id has been successfully created. It contains one layer of versioned type, so it stores slowly changing data that must remain logically consistent with other layers in the catalog.

This layer has the ID example-layer-id and the following properties:

volumeType: durable Describes storage details of the layer. This layer has durable storage volume type because in versioned layers data is unchanged in the scope of one version.
billingTags: [] List of billing tags - a free-form tag that is used for grouping billing records together. For this layer, billing tags are not present.
partitioningScheme: generic Defines the structure and encoding of the data by associating a schema with the layer. For this layer, the partitioning scheme is set to generic, which is the simplest form of partitioning. In this case, partition names have no semantic meaning. For more information, see the Partitions documentation.
contentType: text/csv The MIME type of the data that is stored in this layer. MIME type text/csv has been created for the handling of text-based fields.

For more information on layers, see the Data User Guide.

Upload data to a catalog

Create a file with the details you want to upload, such as partition-0.csv in the following example.

    City,Country
    Berlin,Germany
    Seattle,USA
    Chicago,USA
    Boston,USA

Enter the olp catalog layer partition put command.

    olp catalog layer partition put hrn:here:data::org:first-catalog-example-id \
     example-layer-id --partitions testPart0:<path_to_file>/partition-0.csv

    olp catalog layer partition put hrn:here:data::org:first-catalog-example-id ^
    example-layer-id --partitions testPart0:<path_to_file>\partition-0.csv

The command returns the following result:

    Uploading the files into partitions.

     100% [=======================================] 1/1 (0:00:01 / 0:00:00)
    The data was successfully added to the partitions: [testPart0]

To verify the partition, enter the olp catalog layer partition list command.

olp catalog layer partition list hrn:here:data::org:first-catalog-example-id example-layer-id

The console displays the following text:

    partition           size                checksum
    testPart0           23                  665d2817c51cbccd4312eb22699163c05b9d5a0a

    Total size: 23 B
    Use olp catalog layer partition get <catalog HRN> <layer ID> to download partitions

To retrieve the partition-0.csv file from the olpcli-example-catalog-id, enter the olp catalog layer partition get command.

    olp catalog layer partition get \
     hrn:here:data::org:first-catalog-example-id example-layer-id --partitions testPart0

    olp catalog layer partition get ^
    hrn:here:data::org:first-catalog-example-id example-layer-id --partitions testPart0

The console displays the text stored in the partition-0.csv file:

    City,Country
    Berlin,Germany
    Seattle,USA
    Chicago,USA
    Boston,USA

To get a full list of the available commands, enter olp --help.

Updated 21 days ago