GuidesChangelogTypeScript API ReferencePython v2 API Reference

here.platform.partition module

here.platform.partition module

HERE platform catalog layer partition abstraction.

class here.platform.partition.IndexPartition(layer: IndexLayer, data_handle: str | None = None, checksum: str | None = None, data_size: int | None = None, crc: str | None = None, timestamp: int | None = None, fields: Dict[str, str | int | bool] = , additional_metadata: Dict[str, str] = )
Bases: Partition

Partition subclass used for ``IndexLayer``s.

In addition to the fields present in the base class, this class adds: - timestamp: timestamp of the partition - fields: fields according to the index layer configuration - additional_metadata: free-form additional metadata

class here.platform.partition.Partition(data_handle: str | None = None, layer=None, id: str | None = None, checksum: str | None = None, data_size: int | None = None, crc: str | None = None)
Bases: object

HERE platform partition abstraction.

static generate_data_handle() → str
Generate a unique data handle.

Returns:
A string representing a unique blob identifier

get_blob(stream: bool = False, chunk_size: int = 102400) → bytes | Iterator[bytes]

Get blob (raw bytes) inside the partition for this layer with given data_handle.

Parameters:

  • stream – whether to stream data.

  • chunk_size – the size to request each iteration when streaming data.

Returns:
Content of the blob referenced by the data handle.

Raises:
ValueError – Unsupported content encoding or invalid data handle

static get_data_handler(path_or_data: str | bytes | Path)
Get the data handler to read from file path or data.

Parameters:
path_or_data – Path of the file or data to be read and uploaded as a partition.

Yield Union[BytesIO, BinaryIO]:
Data Handle

multipart_upload(path_or_data: str | bytes | Path, part_size: int)

Multipart upload data to a blob store.

Parameters:

  • path_or_data – Path of the file or data in bytes to be upload as a partition.

  • part_size – An int representing size in MB, to upload in multiple parts minimum value is 5MB and and Maximum is 50MB.

Raises:
ValueError – Data handle not generated, max retry timeout reached

put_blob(path_or_data: str | bytes | Path)

Upload data in single part.

Parameters:
path_or_data – Path of the file or data in bytes to be uploaded as a partition.

async read_and_upload_parts(path_or_data: str | bytes | Path, upload_url: str, content_type: str, headers: Dict[str, str], part_size: int)
Read input file in chunks and upload each part to the specified URL.

Parameters:

  • path_or_data – Path of the file or data to be read and upload as a partition.

  • upload_url – An URL to upload blob data.

  • content_type – A standard MIME type describing the format of the blob data.

  • headers – A dict containing http headers.

  • part_size – An int representing size in MB, to upload in multiple parts minimum value is 5MB and and Maximum is 50MB.

Returns:
A list of dict with key as part number and value as url response body.

async upload_part(session: ClientSession, upload_url: str, part_num: int, headers: Dict[str, str], data: bytes) → Dict
Upload a part data to a url, using specified ClientSession.

Parameters:

  • session – a ClientSession object for making requests.

  • upload_url – an URL to upload blob data.

  • part_num – an unique number for the part to be uploaded.

  • headers – HTTP headers to be send with the request.

  • data – blob data to be send with the request.

Returns:
Returns a dict with key as part number and value as response body.

upload_volatile_blob(path_or_data: str | Path | bytes)

Upload data to volatile blob.

Parameters:
path_or_data – Path of the file or data to be read and upload as a partition

Raises:
ValueError – in case data handle is not set

class here.platform.partition.StreamPartition(layer: StreamLayer, data_handle: str | None = None, id: str | None = None, checksum: str | None = None, data_size: int | None = None, crc: str | None = None, data: bytes | None = None, timestamp: int | None = None, kafka_partition: int | None = None, kafka_offset: int | None = None)
Bases: Partition

Partition subclass used for ``StreamLayer``s.

In addition to the fields present in the base class, this class adds: - data: inline data, alternative to data_handle - timestamp: timestamp in milliseconds since the Unix epoch (1970-01-01T00:00:00 UTC) - kafka_offset: the offset of the message in the Kafka stream partition - kafka_partition: the Kafka stream partition number the offset is related to

get_data(stream: bool = False, chunk_size: int = 102400) → bytes | Iterator[bytes]

Get the data associated with the StreamPartition.

Data can be present directly in the data field or retrieved via data_handle. This function returns the data, regardless of where it is stored.

Returns:
Data associated with the stream partition

Parameters:

  • stream – whether to stream data

  • chunk_size – the size to request each iteration when streaming data

Raises:
ValueError – in case neither in data nor data_handle is available

class here.platform.partition.VersionedPartition(data_handle: str | None, layer: VersionedLayer, id: str, checksum: str | None = None, data_size: int | None = None, compressed_data_size: int | None = None, crc: str | None = None, version: int | None = None)
Bases: Partition

Partition subclass used for ``VersionedLayer``s.

In addition to the fields present in the base class, this class adds: - version: version of the partition

class here.platform.partition.VolatilePartition(data_handle: str | None, layer: VolatileLayer, id: str | None = None, checksum: str | None = None, data_size: int | None = None, compressed_data_size: int | None = None, crc: str | None = None)
Bases: Partition

Partition subclass used for ``VolatileLayer``s.

here.platform.models module | here.platform.platform module

Show Source