Read to DataFrame

To read data and metadata from supported layer types into a DataFrame or GeoDataFrame, enable the GeoPandasAdapter and use the standard HERE Data SDK read functions.

To read data and metadata from versioned, volatile, index, stream and interactive map layers, please familiarize yourself first with the read functions described in the corresponding section of this user guide.

All the standard parameters of get_partitions_metadata, read_partitions, read_stream_metadata, read_stream, get_features, iter_features are supported, in addition to adapter-specific parameters that are forwarded to this adapter and its data decoder.

When reading and decoding data, parameters that are adapter-specific are passed to the pd.read_csv, pd.read_parquet and similar Pandas functions that perform the actual decoding of each single partition. You can use them to fine-tune the details of the decoding of single partitions, including how to handle the (Geo)DataFrame index, if present in the data.

The GeoPandasAdapter puts together the output in a single DataFrame. For more information on supported content types and exact parameters, please see the documentation of GeoPandasDecoder. The partition name is saved in a partition_id column, to distinguish data read from one partition from data read from another partition, when reading multiple partitions at once. The actual name of the partition_id column can be configured in the GeoPandasAdapter constructor, together with other parameters to fine-tune decoding of specific formats like content following a Protocol Buffers schema.

In case decode=False is passed to read_partitions or read_stream, no decoding takes places, the adapter is not used and a plain Python collection or iterator containing bytes is returned.

Get partitions data and metadata from versioned layer in a DataFrame

Use get_partitions_metadata to obtain partitions metadata. When the GeoPandasAdapter is enabled, a pd.DataFrame is returned instead of a list or dict as shown in the example below.

Example: getting versioned metadata in a DataFrame

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())
sdii_catalog = platform.get_catalog("hrn:here:data::olp-here:olp-sdii-sample-berlin-2")
versioned_layer = sdii_catalog.get_layer("sample-versioned-layer")

partitions_df = versioned_layer.get_partitions_metadata([377894434, 377894435, 377894440, 377894441])

Partitions metadata are returned in a DataFrame that is not indexed.

Use read_partitions to fetch and decode the data. When the GeoPandasAdapter is enabled, a pd.DataFrame or a gpd.GeoDataFrame, depending on the content, is returned instead of a list or dict as shown in the example below.

Example: reading versioned data in a DataFrame

partitions_df = versioned_layer.read_partitions(partition_ids=[377894434, 377894435])

Partitions data are returned in a DataFrame that is not indexed. Only one pd.DataFrame or gpd.GeoDataFrame is returned. Data of multiple partitions are all included in the same output. A partition_id column is added to disambiguate. The name of the columns depends on the content type, schema and actual content of the layer. If no partition_ids are provided, the whole layer is read.

This specific example reads content encoded in Protobuf format.

(text truncated for clarity)

Depending on the content type and actual schema, the returned DataFrame may be directly usable or require further manipulation to bring it to a usable form. CSV, GeoJSON, Parquet and schemaless content types are decoded and converted to the best possible format for the user automatically. For example, GeoJSON is decoded into a gpd.GeoDataFrame. Protobuf-encoded data usually have nested, composite and repeated fields, lists, dictionaries, and other complex data structures.

Documentation of GeoPandasDecoder illustrates parameters that can be used to fine-tune the decoding and improve the resulting output for every content type, but in particular for Protobuf-encoded data. Very common is the record_path parameter: when specified, only content in that path is decoded. If the field at the given path happens to be a repeated field, the function returns multiple rows per partition. Dictionaries are also unpacked automatically to multiple columns, when possible.

Continuing the example above, we read again the same partitions, specifying the record_path parameter and selecting only some columns for clarity:

columns = ["messageId", "message.envelope.transientVehicleUUID", "message.path.positionEstimate", "metadata.receivedTime"]

messages_df = versioned_layer.read_partitions(partition_ids=[377894434, 377894435], record_path="messages", columns=columns)

results in:

(text and rows truncated for clarity)

The partition_id columns is always added automatically after decoding.

The column message.path.positionEstimate contains a list, that can be further processed, turning the DataFrame from having one row per message to one row per position estimate:

from here.geopandas_adapter.utils.dataframe import unpack_columns

estimates_df = messages_df[["messageId", "message.path.positionEstimate"]].explode("message.path.positionEstimate")
estimates_df = unpack_columns(estimates_df, "message.path.positionEstimate", keep_prefix=False)

results in:

(columns and rows truncated for clarity)

Get partitions data and metadata from volatile layer in a DataFrame

Use get_partitions_metadata to obtain partitions metadata. When the GeoPandasAdapter is enabled, a pd.DataFrame is returned instead of a list or dict as shown in the example below.

Example: getting volatile metadata in a DataFrame

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())
weather_catalog = platform.get_catalog('hrn:here:data::olp-here:live-weather-eu')
volatile_layer = weather_catalog.get_layer('latest-data')

partitions_df = volatile_layer.get_partitions_metadata(partition_ids=[81150, 81151])

Partitions metadata are returned in a DataFrame that is not indexed.

Use read_partitions to fetch and decode the data. When the GeoPandasAdapter is enabled, a pd.DataFrame or a gpd.GeoDataFrame, depending on the content, is returned instead of a list or dict as shown in the example below.

Note

Volatile metadata and underlying data can occasionally be out of sync. When this occurs, metadata may indicate that data exists in a given partition but at the current point in time there is no data residing there. In the event you call read_partitions and one or more of the requested partitions do not exist or contain no data, no rows will be added to the returned DataFrame for that partition. This could result in an empty DataFrame being returned.

Example: reading volatile data in a DataFrame

partitions_df = volatile_layer.read_partitions(partition_ids=[81150, 81151], record_path="weather_condition_tile")

Partitions data are returned in a DataFrame that is not indexed. Only one pd.DataFrame or gpd.GeoDataFrame is returned. Data of multiple partitions are all included in the same output. A partition_id column is added to disambiguate. The name of the columns depends on the content type, schema and actual content of the layer. If no partition_ids are provided, the whole layer is read.

This specific example reads content encoded in Protobuf format.

columns = ["tile_id",
           "center_point_geohash",
           "air_temperature.value",
           "dew_point_temperature.value",
           "humidity.value",
           "air_pressure.value",
           "visibility.value",
           "iop.value",
           "wind_velocity.value",
           "wind_velocity.direction",
           "precipitation_type.precipitation_type"]

partitions_df = volatile_layer.read_partitions(partition_ids=[81150, 81151], record_path="weather_condition_tile", columns=columns)

In this example we select only some columns obtained from the Protobuf repeated field weather_condition_tile, resulting in:

(rows truncated for clarity)

Get partitions data and metadata from index layer in a DataFrame

Use get_partitions_metadata to obtain partitions metadata. When the GeoPandasAdapter is enabled, a pd.DataFrame is returned instead of a list or dict as shown in the example below.

Example: getting index metadata in a DataFrame

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())
sdii_catalog = platform.get_catalog("hrn:here:data::olp-here:olp-sdii-sample-berlin-2")
index_layer = sdii_catalog.get_layer("sample-index-layer")

partitions_df = index_layer.get_partitions_metadata(query="hour_from=ge=10")

Partitions metadata are returned in a DataFrame that is not indexed. The data handle is used in place of partition id, since the index layer doesn't have a proper identifier for partitions.

Use read_partitions to fetch and decode the data. When the GeoPandasAdapter is enabled, a pd.DataFrame or a gpd.GeoDataFrame, depending on the content, is returned instead of a list or dict as shown in the example below. If no partition_ids are provided, the whole layer is read.

Example: reading index data in a DataFrame

partitions_df = index_layer.read_partitions(query="hour_from=ge=10")

Partitions data are returned in a DataFrame that is not indexed. Only one pd.DataFrame or gpd.GeoDataFrame is returned. Data of multiple partitions are all included in the same output. A partition_id column is added to disambiguate. The name of the columns depends on the content type, schema and actual content of the layer. The data handle is used in place of partition id, since the index layer doesn't have a proper identifier for partitions.

(text and rows truncated for clarity)

In this specific example, as demonstrate for other layer types and described in details in the section Manipulate DataFrames and GeoDataFrames, it's convenient to use the unpack_columns function to further unpack the dictionaries into proper columns:

from here.geopandas_adapter.utils import dataframe

columns = ["partition_id", "pathEvents"]

events_df = dataframe.unpack_columns(partitions_df[columns], ["pathEvents"], keep_prefix=False)

resulting in:

(text, columns and rows truncated for clarity)

Get partitions data and metadata from stream layer in a DataFrame

Use get_stream_metadata to consume partitions metadata from a stream subscription. When the GeoPandasAdapter is enabled, a pd.DataFrame is returned instead of a list or dict as shown in the example below.

Example: getting stream metadata in a DataFrame

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())
sdii_catalog = platform.get_catalog("hrn:here:data::olp-here:olp-sdii-sample-berlin-2")
stream_layer = sdii_catalog.get_layer("sample-streaming-layer")

with stream_layer.subscribe() as subscription:
    partitions_df = stream_layer.get_stream_metadata(subscription=subscription)

Partitions metadata (stream messages) are returned in a DataFrame that is not indexed. Data can be inlined, as in this example, or stored via the Blob API if too large.

(text and rows truncated for clarity)

Use read_stream to consume, fetch and decode the data from a stream subscription. When the GeoPandasAdapter is enabled, a pd.DataFrame or a gpd.GeoDataFrame, depending on the content, is returned instead of a list or dict as shown in the example below.

Example: reading stream data in a DataFrame

In this example we show how adapter-specific parameters, such as record_path, can be used to customize the decoding. We're interested in only a selection of the properties of the data.

This specific example reads content encoded in Protobuf format.

with stream_layer.subscribe() as subscription:
    columns = ["timeStampUTC_ms",
               "latitude_deg",
               "longitude_deg",
               "heading_deg",
               "speed_mps"]

    partitions_df = stream_layer.read_stream(subscription=subscription, record_path="path.positionEstimate", columns=columns)

Partitions data are returned in a DataFrame that is not indexed. Only one pd.DataFrame or gpd.GeoDataFrame is returned. Data of multiple partitions are all included in the same output. A partition_id column is added to disambiguate. The name of the columns depends on the content type, schema and actual content of the layer.

(rows truncated for clarity)

Get features from interactive map layer in a GeoDataFrame

Use search_features to retrieve features from one interactive map layer. When the GeoPandasAdapter is enabled, a gpd.GeoDataFrame is returned instead of a list or dict as shown in the example below.

The layer supports other functions, among which get_features and spatial_search that query and retrieve features from the layer. A GeoDataFrame is returned from these functions as well.

When running in Jupyter notebooks, a GeoDataFrame enables an effortless, visual inspection of the features over a map, as demonstrated by using the HERE Inspector in the examples below.

Example: reading features in a GeoDataFrame

In this example we retrieve the districts (Bezirk) of Berlin from a sample catalog and a sample interactive map layer.

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())

sample_catalog = platform.get_catalog("hrn:here:data::olp-here:here-geojson-samples")
iml_layer = sample_catalog.get_layer("berlin-interactivemap")

features_gdf = iml_layer.search_features()

search_features without parameters returns all the content, resulting in:

(text and rows truncated for clarity)

It's also possible to specify search parameters, as in the following case:

features_gdf = iml_layer.search_features(params={"p.BezName": "Pankow"}, force_2d=True)

resulting in the selection of just one district and removal of z-level from the coordinates:

(text truncated for clarity)

Result can be rendered directly on a map when running in a Jupyter notebook, for example using the doc:here-inspectorHERE Inspector:

from here.inspector import inspect
from here.inspector.styles import Color

inspect(features_gdf, "Districts of Berlin", style=Color.BLUE)

Example: geospatial search of features in a GeoDataFrame

In this example we query the districts of Berlin within a 1000m-distance from a city landmark, the Zoologischer Garten railway station, located at the coordinates visible in the query.

from here.platform import Platform
from here.geopandas_adapter import GeoPandasAdapter

platform = Platform(adapter=GeoPandasAdapter())

sample_catalog = platform.get_catalog("hrn:here:data::olp-here:here-geojson-samples")
iml_layer = sample_catalog.get_layer("berlin-interactivemap")

features_gdf = iml_layer.spatial_search(lng=13.33474, lat=52.50686, radius=1000)

resulting in:

The result can be rendered directly in a Jupyter notebook using:

from here.inspector import inspect
from here.inspector.styles import Color

inspect(features_gdf, "Districts within 1000m from Berlin Zoologischer Garten railway station", style=Color.RED)