How to use Pandas and GeoPandas

A GeoPandasAdapter is provided in package here-geopandas-adapter to ease working with Pandas and GeoPandas libraries. Once imported, instantiated and enabled in the Platform, many read and write functions of the HERE Data SDK for Python accept and return pd.DataFrame, pd.GeoSeries, gpd.GeoDataFrame and gpd.GeoSeries in place of Python list and dict objects.

• Enabling the adapter
• Read to DataFrame
• Write DataFrame to layer
• Manipulate DataFrames

The HERE GeoPandas Adapter can be applied in any of three ways:

• to all read/write operations
• on a per-catalog basis
• on a per-function-call basis

Below we illustrate these three options.

To have the adapter apply to all catalogs and other entities created through a Platform object you can specify adapter when instantiating that Platform object:

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())

It's also possible to enable the adapter only for selected catalogs, specifying it in the corresponding get_catalog call:

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform()
adapter = GeoPandasAdapter()

# These catalogs use the adapter
weather_eu = platform.get_catalog('hrn:here:data::olp-here:live-weather-eu', adapter=adapter)
weather_na = platform.get_catalog('hrn:here:data::olp-here:live-weather-na', adapter=adapter)

# This catalog does not
sdii = platform.get_catalog("hrn:here:data::olp-here:olp-sdii-sample-berlin-2")

Lastly, it's also possible to specify the use of the adapter in single functions:

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform()
adapter = GeoPandasAdapter()

weather_na = platform.get_catalog('hrn:here:data::olp-here:live-weather-na')
live_layer = weather_na.get_layer('latest-data')

# This function uses the adapter
weather_df = live_layer.read_partitions([75477, 75648, 75391, 75562], adapter=adapter, record_path="weather_condition_tile")

# This function does not
weather_msgs = live_layer.read_partitions([75477, 75648, 75391, 75562])

When reading and decoding data, parameters that are adapter-specific are passed to the pd.read_csv, pd.read_parquet and similar Pandas functions that perform the actual decoding of each single partition. You can use them to fine-tune the details of the decoding of single partitions, including how to handle the (Geo)DataFrame index, if present in the data.

Use get_partitions_metadata to obtain partitions metadata. When the GeoPandasAdapter is enabled, a pd.DataFrame is returned instead of a list or dict as shown in the example below. Example: getting versioned metadata in a DataFrame

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())
sdii_catalog = platform.get_catalog("hrn:here:data::olp-here:olp-sdii-sample-berlin-2")
versioned_layer = sdii_catalog.get_layer("sample-versioned-layer")

partitions_df = versioned_layer.get_partitions_metadata([377894434, 377894435, 377894440, 377894441])

Partitions metadata are returned in a DataFrame that is not indexed.

partitions_df = versioned_layer.read_partitions(partition_ids=[377894434, 377894435])

Partitions data are returned in a DataFrame that is not indexed. Only one pd.DataFrame or gpd.GeoDataFrame is returned. Data of multiple partitions are all included in the same output. A partition_id column is added to disambiguate. The name of the columns depends on the content type, schema and actual content of the layer. If no partition_ids are provided, the whole layer is read.

This specific example reads content encoded in Protobuf format.

columns = ["messageId", "message.envelope.transientVehicleUUID", "message.path.positionEstimate", "metadata.receivedTime"]

messages_df = versioned_layer.read_partitions(partition_ids=[377894434, 377894435], record_path="messages", columns=columns)

results in:

from here.geopandas_adapter.utils.dataframe import unpack_columns

estimates_df = messages_df[["messageId", "message.path.positionEstimate"]].explode("message.path.positionEstimate")
estimates_df = unpack_columns(estimates_df, "message.path.positionEstimate", keep_prefix=False)

results in:

Get partitions data and metadata from volatile layer in a DataFrame

Use get_partitions_metadata to obtain partitions metadata. When the GeoPandasAdapter is enabled, a pd.DataFrame is returned instead of a list or dict as shown in the example below. Example: getting volatile metadata in a DataFrame

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())
weather_catalog = platform.get_catalog('hrn:here:data::olp-here:live-weather-eu')
volatile_layer = weather_catalog.get_layer('latest-data')

partitions_df = volatile_layer.get_partitions_metadata(partition_ids=[81150, 81151])

Partitions metadata are returned in a DataFrame that is not indexed.

📘

Note

Volatile metadata and underlying data can occasionally be out of sync. When this occurs, metadata may indicate that data exists in a given partition but at the current point in time there is no data residing there. In the event you call read_partitions and one or more of the requested partitions do not exist or contain no data, no rows will be added to the returned DataFrame for that partition. This could result in an empty DataFrame being returned.

Example: reading volatile data in a DataFrame

partitions_df = volatile_layer.read_partitions(partition_ids=[81150, 81151], record_path="weather_condition_tile")

Partitions data are returned in a DataFrame that is not indexed. Only one pd.DataFrame or gpd.GeoDataFrame is returned. Data of multiple partitions are all included in the same output. A partition_id column is added to disambiguate. The name of the columns depends on the content type, schema and actual content of the layer. If no partition_ids are provided, the whole layer is read.

This specific example reads content encoded in Protobuf format.

columns = ["tile_id",
           "center_point_geohash",
           "air_temperature.value",
           "dew_point_temperature.value",
           "humidity.value",
           "air_pressure.value",
           "visibility.value",
           "iop.value",
           "wind_velocity.value",
           "wind_velocity.direction",
           "precipitation_type.precipitation_type"]

partitions_df = volatile_layer.read_partitions(partition_ids=[81150, 81151], record_path="weather_condition_tile", columns=columns)

In this example we select only some columns obtained from the Protobuf repeated field weather_condition_tile, resulting in:

Get partitions data and metadata from index layer in a DataFrame

Use get_partitions_metadata to obtain partitions metadata. When the GeoPandasAdapter is enabled, a pd.DataFrame is returned instead of a list or dict as shown in the example below. Example: getting index metadata in a DataFrame

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())
sdii_catalog = platform.get_catalog("hrn:here:data::olp-here:olp-sdii-sample-berlin-2")
index_layer = sdii_catalog.get_layer("sample-index-layer")

partitions_df = index_layer.get_partitions_metadata(query="hour_from=ge=10")

Partitions metadata are returned in a DataFrame that is not indexed. The data handle is used in place of partition id, since the index layer doesn't have a proper identifier for partitions.

partitions_df = index_layer.read_partitions(query="hour_from=ge=10")

Partitions data are returned in a DataFrame that is not indexed. Only one pd.DataFrame or gpd.GeoDataFrame is returned. Data of multiple partitions are all included in the same output. A partition_id column is added to disambiguate. The name of the columns depends on the content type, schema and actual content of the layer. The data handle is used in place of partition id, since the index layer doesn't have a proper identifier for partitions.

from here.geopandas_adapter.utils import dataframe

columns = ["partition_id", "pathEvents"]

events_df = dataframe.unpack_columns(partitions_df[columns], ["pathEvents"], keep_prefix=False)

resulting in:

Get partitions data and metadata from stream layer in a DataFrame

Use get_stream_metadata to consume partitions metadata from a stream subscription. When the GeoPandasAdapter is enabled, a pd.DataFrame is returned instead of a list or dict as shown in the example below. Example: getting stream metadata in a DataFrame

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())
sdii_catalog = platform.get_catalog("hrn:here:data::olp-here:olp-sdii-sample-berlin-2")
stream_layer = sdii_catalog.get_layer("sample-streaming-layer")

with stream_layer.subscribe() as subscription:
    partitions_df = stream_layer.get_stream_metadata(subscription=subscription)

Partitions metadata (stream messages) are returned in a DataFrame that is not indexed. Data can be inlined, as in this example, or stored via the Blob API if too large.

with stream_layer.subscribe() as subscription:
    columns = ["timeStampUTC_ms",
               "latitude_deg",
               "longitude_deg",
               "heading_deg",
               "speed_mps"]

    partitions_df = stream_layer.read_stream(subscription=subscription, record_path="path.positionEstimate", columns=columns)

Partitions data are returned in a DataFrame that is not indexed. Only one pd.DataFrame or gpd.GeoDataFrame is returned. Data of multiple partitions are all included in the same output. A partition_id column is added to disambiguate. The name of the columns depends on the content type, schema and actual content of the layer.

Get features from interactive map layer in a GeoDataFrame

Use search_features to retrieve features from one interactive map layer. When the GeoPandasAdapter is enabled, a gpd.GeoDataFrame is returned instead of a list or dict as shown in the example below.

The layer supports other functions, among which get_features and spatial_search that query and retrieve features from the layer. A GeoDataFrame is returned from these functions as well.

When running in Jupyter notebooks, a GeoDataFrame enables an effortless, visual inspection of the features over a map, as demonstrated by using the HERE Inspector in the examples below.

Example: reading features in a GeoDataFrame In this example we retrieve the districts (Bezirk) of Berlin from a sample catalog and a sample interactive map layer.

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())

sample_catalog = platform.get_catalog("hrn:here:data::olp-here:here-geojson-samples")
iml_layer = sample_catalog.get_layer("berlin-interactivemap")

features_gdf = iml_layer.search_features()

search_features without parameters returns all the content, resulting in:

features_gdf = iml_layer.search_features(params={"p.BezName": "Pankow"}, force_2d=True)

resulting in the selection of just one district and removal of z-level from the coordinates:

from here.inspector import inspect
from here.inspector.styles import Color

inspect(features_gdf, "Districts of Berlin", style=Color.BLUE)

Example: geospatial search of features in a GeoDataFrame In this example we query the districts of Berlin within a 1000m-distance from a city landmark, the Zoologischer Garten railway station, located at the coordinates visible in the query.

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())

sample_catalog = platform.get_catalog("hrn:here:data::olp-here:here-geojson-samples")
iml_layer = sample_catalog.get_layer("berlin-interactivemap")

features_gdf = iml_layer.spatial_search(lng=13.33474, lat=52.50686, radius=1000)

resulting in:

from here.inspector import inspect
from here.inspector.styles import Color

inspect(features_gdf, "Districts within 1000m from Berlin Zoologischer Garten railway station", style=Color.RED)

Write DataFrame to layer

To write data and metadata to versioned, volatile, index, stream and interactive map layers, please familiarize yourself first with the write functions described in the corresponding section of this user guide.

For content types supported by the GeoPandas Adapter (see Table), contents of a DataFrame or GeoDataFrame can be encoded and written to layer with a single function. For content types not supported, you will need to pass encode=False and take care of the encoding yourself.

All the standard parameters of set_partitions_metadata, write_partitions, append_stream_metadata, write_stream, write_features, update_features, delete_features are supported, in addition to adapter-specific parameters that are forwarded to this adapter and its data encoder.

When writing and encoding data, the GeoPandasAdapter splits the (Geo)DataFrame to write in partitions according to the partition_id column. Each selection of rows is then encoded and stored as standalone partition. Rows with no partition identifier set are discarded. Parameters that are adapter-specific are passed to the DataFrame.to_csv, DataFrame.to_parquet and similar functions that perform the actual encoding of each single partition. You can use them to fine-tune the details of the encoding of single partitions, including how to handle the (Geo)DataFrame index. For more information on supported content types and exact parameters, please see the documentation of GeoPandasEncoder.

In case encode=False is passed to write_partitions or write_stream, a plain Python collection containing bytes and not a (Geo)DataFrame must be passed as well, as the adapter is not used and no encoding takes place.

Write examples are symmetric to the read examples shown above.

Manipulate DataFrames and GeoDataFrames

The commonly used Pandas and GeoPandas libraries are well documented, and many examples showing how to use them to perform data analysis and manipulation are publicly available. Generally, data is in a tabular representation where each cell of the table contains one value with a defined data type (numeric, string, or other basic type).

Map data and, in general, data stored in a catalog can be highly structured sometimes and follow a complex, nested schema. Dealing with this complexity in Pandas can be difficult. Therefore, the HERE Data SDK for Python includes in the here-geopandas-adapter package utility functions to perform repetitive tasks and manipulate complex DataFrames, in particular DataFrames with columns that contain dictionaries instead of single values.

Unpacking series and DataFrames

Pandas provides the explode function to turn objects of type list contained in a column into multiple rows. Similarly, HERE Data SDK for Python provides the unpack and unpack_columns functions to turn single columns containing dict into multiple columns. This is a convenience function to unpack data structures that sometimes result from reading data from catalogs or working with complex data models.

unpack is applied to a Series containing dict objects, it returns a DataFrame. unpack_columns is applied to a DataFrame to replace one or more column that contain dict objects with multiple columns, one for each field of the dictionaries. Unpacking is also recursive, to deal easily with deeply nested data structures.

Example: unpacking a DataFrame column that contains dictionaries Given the example DataFrame df, derived from structured objects:

import pandas as pd

berlin = {
    "name": "Berlin",
    "location": {
        "longitude": 13.408333,
        "latitude": 52.518611,
        "country": { "name": "Deutschland", "code": "DE" }
    },
    "zip_codes": { "min": 10115, "max": 14199 },
    "population": 3664088
}

paris = {
    "name": "Paris",
    "location": {
        "longitude": 2.351667,
        "latitude": 48.856667,
        "country": { "name": "France", "code": "FR" }
    },
    "zip_codes": { "min": 75001, "max": 75020 },
    "population": 2175601
}

df = pd.DataFrame([berlin, paris])

resulting in:

from here.geopandas_adapter.utils.dataframe import unpack_columns

unpacked_df = unpack_columns(df, columns=["location", "zip_codes"])

resulting in:

Replacing a column with one or more columns

The function replace_column can be used to replace one single column of a DataFrame with one or multiple columns of another DataFrame. Example: replacing one column with a multiple columns Given the example DataFrames df and df2:

import pandas as pd

df = pd.DataFrame({
    "col_A": [11, 31, 41],
    "col_B": [12, 32, 42],
    "col_C": [14, 34, 42]
}, index = [1, 3, 4])

df2 = pd.DataFrame({
    "col_Bx": [110, 130, 140],
    "col_By": [115, 135, 145]
}, index = [1, 3, 4])

resulting in:

from here.geopandas_adapter.utils.dataframe import replace_column

replaced_df = replace_column(df, "col_B", df2)

resulting in:

Adding and removing prefixes to column names

The functions prefix_columns and unprefix_columns are used to add or remove a prefix from the names of selected columns of a DataFrame. A separator . is added between the prefix and column names.

This is useful to group (prefix) related columns of a DataFrame under a common prefix or to remove a lengthy, verbose prefix present in multiple columns (unprefix) to obtain a derived DataFrame that is more comfortable to work with. Example: prefixing columns with common prefix Given the example DataFrame df:

import pandas as pd

df = pd.DataFrame({
    "name": ["Sarah", "Vivek", "Marco"],
    "age": [41, 29, 35],
    "house_nr": ["1492", "34-35", "48A"],
    "road": ["SE 36th Ave", "Seshadri Road", "Via Giosuè Carducci"],
    "city": ["Portland", "Bengaluru", "Milan"],
    "zip": [97214, 560009, 20123],
    "state": ["OR", "KA", pd.NA],
    "country": ["US", "IN", "IT"],
})

resulting in:

from here.geopandas_adapter.utils.dataframe import prefix_columns

prefixed_df = prefix_columns(df, "address", ["house_nr", "road", "city", "zip", "country", "state"])

resulting in:

from here.geopandas_adapter.utils.dataframe import unprefix_columns

unprefixed_df = unprefix_columns(prefixed_df, "address")

resulting in: