Data workflows

The HERE platform organizes data according to three key
concepts: catalog, layer, and partition. The platform hosts a number
of catalogs, each with one or more layers, which are divided into partitions.

For more information on these concepts, see the
Data API.

The OLP CLI provides tools for managing data, such as:

List all catalogs to which you have access
Create a new catalog and layer
Upload data to a catalog

For more details, see catalog commands.

List all catalogs

To list the catalogs to which you have access, enter the
olp catalog list command.

olp catalog list

The command lists the HRNs of the catalogs to which you have access,
each in a new line.

Create a new catalog and layer

Follow the steps below to create a new catalog and layer.

Enter the olp catalog create command.

    olp catalog create first-catalog-example-id first-catalog-example \
    --summary "This catalog contains a sample layer containing data in CSV format"

    olp catalog create first-catalog-example-id first-catalog-example ^
    --summary "This catalog contains a sample layer containing data in CSV format"

To add a layer to your catalog, enter the olp catalog layer add command.

    olp catalog layer add hrn:here:data::org:first-catalog-example-id example-layer-id \
     "Example layer" --versioned --summary "This layer contains simple data in CSV format" \
     --description "A longer description of the layer contents" --content-type=text/csv

    olp catalog layer add hrn:here:data::org:first-catalog-example-id example-layer-id ^
    "Example layer" --versioned --summary "This layer contains simple data in CSV format" ^
    --description "A longer description of the layer contents" --content-type=text/csv

The command above adds a `versioned` layer with the ID `example-layer-id` and name `Example layer` to the `hrn:here:data::org:first-catalog-example-id` catalog.
The content type of this layer is set to `text/csv`.

For `Example layer`, the following parameters were not explicitly used and set to their default values:
- `--content-encoding uncompressed` The content encoding of the layer.
  If this parameter is specified, all data in a layer is compressed by the specified algorithm.
  The default value is `uncompressed`, which means no compression.
- `--partitioning generic` The [partitioning scheme](https://docs.here.com/data-api/docs/partitions)
  of the layer, either `heretile` or `generic`. For `heretile`, you can specify `tile levels` separated by a comma, such as `heretile:level1,level2`.
  The platform supports tile levels from 1 to 15. For more information on `HERE Tile Partitioning`, see the [Calculate Partition Tile IDs tutorial](https://docs.here.com/workspace/docs/tutorials-here-tile-resolver-readme).
  The default value is `generic`.

3. To verify the catalog configuration, enter the
olp catalog show command.

olp catalog show hrn:here:data::org:first-catalog-example-id --json

The command displays the following results.

{
        "summary": "This catalog contains sample layers containing simple text",
        "owner": {
            "creator": {"id": "mzLcb1rL8nskvDQpCFEF"},
            "organisation": {"id": "org"}
        },
        "billingTags": [],
        "hrn": "hrn:here:data::org:first-catalog-example-id",
        "created": "2018-03-26T12:48:10.315Z",
        "name": "first-catalog-example",
        "layers": [{
            "summary": "This layer contains simple data in CSV format",
            "volume": {
                 "volumeType": "durable"
            },
            "layerType": "versioned",
            "billingTags": [],
            "name": "Example layer",
            "description": "A longer description of the layer contents",
            "partitioningScheme": "generic",
            "partitioning": {
                 "scheme": "generic"
            },
            "id": "example-layer-id",
            "contentType": "text/csv",
            "tags": []
        }],
        "description": "",
        "notifications": {
            "enabled": false
         },
        "id": "first-catalog-example-id",
        "version": -1,
        "tags": []
    }

From the command output above, you can see that a catalog with HRN `hrn:here:data::org:first-catalog-example-id` has been successfully created.
It contains one layer of `versioned` type, so it stores slowly changing data that must remain logically consistent with other layers in the catalog.

This layer has the ID `example-layer-id` and the following properties:
- `volumeType: durable` Describes storage details of the layer. This layer has `durable` storage volume type because in `versioned` layers data is unchanged in the scope of one version.
- `billingTags: []` List of billing tags - a free-form tag that is used for grouping billing records together. For this layer, billing tags are not present.
- `partitioningScheme: generic` Defines the structure and encoding of the data by associating a schema with the layer.
  For this layer, the partitioning scheme is set to `generic`, which is the simplest form of partitioning.
  In this case, partition names have no semantic meaning. For more information, see the [Partitions documentation](https://docs.here.com/data-api/docs/partitions).
- `contentType: text/csv` The MIME type of the data that is stored in this layer. MIME type `text/csv` has been created for the handling of text-based fields.

For more information on layers, see the [Data User Guide](https://docs.here.com/data-api/docs/layers).

Upload data to a catalog

Create a file with the details you want to upload, such as
partition-0.csv in the following example.

    City,Country
    Berlin,Germany
    Seattle,USA
    Chicago,USA
    Boston,USA

Enter the
olp catalog layer partition put
command.

    olp catalog layer partition put hrn:here:data::org:first-catalog-example-id \
     example-layer-id --partitions testPart0:<path_to_file>/partition-0.csv

    olp catalog layer partition put hrn:here:data::org:first-catalog-example-id ^
    example-layer-id --partitions testPart0:<path_to_file>\partition-0.csv

The command returns the following result:

    Uploading the files into partitions.

     100% [=======================================] 1/1 (0:00:01 / 0:00:00)
    The data was successfully added to the partitions: [testPart0]

To verify the partition, enter the
olp catalog layer partition list
command.

olp catalog layer partition list hrn:here:data::org:first-catalog-example-id example-layer-id

The console displays the following text:

    partition           size                checksum
    testPart0           23                  665d2817c51cbccd4312eb22699163c05b9d5a0a

    Total size: 23 B
    Use olp catalog layer partition get <catalog HRN> <layer ID> to download partitions

To retrieve the partition-0.csv file from the
olpcli-example-catalog-id, enter the
olp catalog layer partition get
command.

    olp catalog layer partition get \
     hrn:here:data::org:first-catalog-example-id example-layer-id --partitions testPart0

    olp catalog layer partition get ^
    hrn:here:data::org:first-catalog-example-id example-layer-id --partitions testPart0

The console displays the text stored in the `partition-0.csv` file:

    City,Country
    Berlin,Germany
    Seattle,USA
    Chicago,USA
    Boston,USA

To get a full list of the available commands, enter olp --help.