GuidesPipeline API ReferenceChangelog
Guides
  1. Data SDK for Java and Scala
  2. How to Use the HERE Data SDK for Java & Scala

How to include the SDK in your project

How to include the SDK in your project

Applications which run in the HERE Workspace may have dependencies
on the following components:

  • HERE Data SDK for Java & Scala libraries, such as the Location Library, the Data Processing
    Library, or the Data Client Library.

  • Libraries provided as part of a runtime environment. The Workspace supports
    two runtime environments: batch and stream. The batch environment provides a minimal
    amount of libraries that Apache Spark applications
    use. The stream environment provides libraries that the
    Apache Flink applications use.

  • Protobuf schemas for encoding and decoding data, like schemas
    for HERE Map Content, HERE Weather, or those which are user provided.

For all mentioned types of dependencies, the HERE Data SDK provides different methods of managing
these dependencies.

HERE Data SDK libraries

The libraries within the Data SDK have different versions. For example,
Data Processing Library 3.1.13 and Data Client Library 0.2.24 are part of the same
Data SDK release. Also, there are interdependencies within the Data SDK, for example
the Location Library and the Data Processing Library depend on the Data Client Library, which
you must consider when managing dependencies.

To help with dependency management, the Data SDK provides Bill Of Materials (BOMs) files.
Each BOM is a Maven POM file which lists compatible versions of these libraries.

There are three different BOM files:

  • Scala 2.13:
    • sdk-batch-bom_2.13.pom - This file contains compatible versions of the HERE Data SDK for Java & Scala libraries,
      their dependencies and the libraries provided by the HERE Workspace Batch 5.0.0 runtime
      environment (Apache Spark 4.0.1).
      See SDK Libraries for Batch Processing with Scala 2.13.
    • sdk-stream-bom_2.13.pom - This file contains compatible versions of Data SDK libraries,
      their dependencies, and the libraries provided by the HERE Workspace Stream 6.0 runtime
      environment (Apache Flink 1.19.2).
      See SDK Libraries for Stream Processing with Scala 2.13.
    • sdk-standalone-bom_2.13.pom - This file contains compatible libraries of Data SDK
      and their dependencies. Use this POM file for running the application on your
      own environment.
      See SDK Libraries for usage in Standalone mode with Scala 2.13.

Notice that you need repository credentials to resolve dependencies.

Include BOMs

You can include the BOM files in your project POM file in two different ways:

  1. As parent POM file (recommended):
 
<parent>
    <groupId>com.here.platform</groupId>
    <artifactId>sdk-batch-bom_2.13</artifactId>
    <version>2.88.6</version>
</parent>
  1. Or, if your project already has a parent, use import for a dependency:
<dependencyManagement>
    <dependencies>
        <dependency>
            
            <groupId>com.here.platform</groupId>
            <artifactId>sdk-batch-bom_2.13</artifactId>
            <version>2.88.6</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

Note

Examples
All code snippets use sdk-batch-bom_2.13 as an example, but the same instructions
are applicable to sdk-stream-bom_2.13 and sdk-standalone-bom_2.13.
Be aware that only one SDK BOM file should be used in a project.

We recommend including the BOM files as a parent because the files contain a
plugin management section and useful properties.

For example, the plugin management section of sdk-batch-bom_2.13 and sdk-stream-bom_2.13
contains the platform profile to create a fat JAR. You need to upload this fat
JAR to the Workspace before you run the application in the cloud.

Note

The platform profile excludes some files and shades some libraries to ensure proper operation when using the JAR
from within a Pipeline.
In particular, the application.properties file is excluded from the platform profile
in the sdk-stream-bom_2.13 file to not override the application.properties file provided by the runtime.

Another important part of the plugin management section and platform profile
is maven-shade-plugin which shades protobuf-java 3+. The shading mechanism
is required because of a conflict between the version of the protobuf-java
library used in Apache Flink and Apache Spark, and the version of this library
used by Data SDK libraries and Protobuf layer schemas. The Data SDK uses a newer
version 3+, but both stream and batch environments provide protobuf 2+.

You can activate the platform profile as follows:
mvn -Pplatform package or mvn --activate-profiles=platform package.

If you cannot include the files as parent into your POM,
check the content of the files, copy properties, and plugin management
configuration which are relevant for your projects.

Include dependencies

As soon as your project includes a BOM file, the project can reference a Data SDK library
or dependency without specifying the actual version of the library.


<dependencies>
    
    <dependency>
        <groupId>com.here.platform.data.processing</groupId>
        <artifactId>pipeline-runner-java_2.13</artifactId>
    </dependency>
</dependencies>

This approach simplifies migration from one Data SDK version to a newer one. Changing
the version of the sdk-batch-bom_2.13 updates the versions of all Data SDK libraries
and their dependencies, and the versions of the libraries provided by the batch environment.

This approach simplifies the migration from one Data SDK version to a more recent one. If
you change the version of the sdk-batch-bom_2.13 in your project, your project uses
updated versions of the following:

  • Data SDK libraries
  • dependencies of all Data SDK libraries
  • libraries provided by the batch environment

When an application which depends on a different version of a
package is defined in the BOM, it may see runtime exceptions such as the following:

  • ClassNotFoundException
  • NoSuchMethodError

These exceptions are raised when classes or methods are not found in the library provided
in the runtime environment. There are two common cases:

  1. The runtime environment contains an older version of the library, and the application
    requires a newer one with new methods and classes.
  2. The runtime environment contains a newer version of the library, but the application
    uses removed deprecated methods or classes from an old version.

If the library is referenced in your project, you can use a corresponding
version of the library from the BOM files. However, the version of one of
the transitive dependencies may have a conflict with the version provided in the runtime environment.
In this case, you need to shade this transitive dependency. For an example of
how to shade a library, see the platform profile in sdk-batch-bom_2.13 or sdk-stream-bom_2.13.
This profile shades the protobuf-java package as mentioned earlier.

For more information about how to import dependencies and BOMs, see the
Maven Dependency Management Guide.

Note

Archetypes
The Workspace-provided
batch
and
streaming
archetypes contain BOMs and have the necessary dependencies in their project
POM files. Only the Maven build system supports these archetypes.

BOM files in SBT projects

SBT does not have built-in support for BOM files, but you can use
the here-sbt-bom plugin.

To use Data SDK libraries in the SBT project, use the following procedure:

  1. Add the MAVEN_CENTRAL repository using a special pattern configuration in the plugins.sbt file:
resolvers += Resolver.url(
"MAVEN_CENTRAL",
url("https://repo.maven.apache.org/maven2"))(
Patterns("[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]", 
"[organisation]/[module]/[revision]/[artifact]_[scalaVersion]_[sbtVersion]-[revision](-[classifier]).[ext]") )

This will enable the project to access plugin files stored
in Maven Central.

  1. Add the here-sbt-bom plugin to plugins.sbt file.
addSbtPlugin("com.here.platform" %% "sbt-bom" % "1.0.33")
  1. Create a file with credentials at ~/.sbt/.credentials with the following content:
realm=Artifactory Realm
host=repo.platform.here.com
user=yourUserId
password=topSecretPassword
  1. Include the credentials in the build.sbt file
project.settings(
credentials += Credentials(Path.userHome / ".sbt" / ".credentials")
)
  1. Include the HERE platform repository resolver
ThisBuild / resolvers += (
"HERE Platform Repository" at "https://repo.platform.here.com/artifactory/open-location-platform"
)
  1. Create a Dependencies class that takes a Bom as a constructor parameter. Inside this class, we can refer to the
    versions from the BOM. The class should expose a list of dependencies that are used as libraryDependencies
import sbt._
import com.here.bom.Bom

case class Dependencies(bom: Bom) {
val dependencies: Seq[ModuleID] = Seq(
"com.here.platform.data.processing" %% "pipeline-runner" % bom
)
}
  1. Add BOM dependency to the build.sbt
import com.here.bom.Bom
lazy val sdkBom = Bom.read("com.here.platform" %% "sdk-batch-bom" % "2.88.6")(bom =>
Dependencies(bom))
  1. Enable this setting in the module configuration and use this setting in libraryDependencies:
lazy val `root` = project
.in(file("."))
.settings(sdkBom)
.settings(
libraryDependencies ++= sdkBom.key.value.dependencies
)

After all steps are completed you can verify that all are configured correctly by running a command in a project root
directory:

sbt package

In case of success, the output looks like:

[success] Total time: 4 s

BOM files in Gradle projects

Gradle provides support for importing BOM files
to control the dependency versions of direct and transitive dependencies.

To use Data SDK libraries in the Gradle project, use the following procedure:

  1. Include the Maven Central and HERE platform repositories. For the HERE Platform repository you need to provide the username and password that can be downloaded from the Platform Portal:
repositories {
    mavenCentral()
    maven {
        url "https://repo.platform.here.com/artifactory/open-location-platform/"
        credentials {
              username = System.getenv("REPO_USERNAME") ?: "defaultUsername"
              password = System.getenv("REPO_PASSWORD") ?: "defaultPassword"
        }
    }
}
  1. Add BOM dependency to the build.gradle file:
dependencies {
implementation platform('com.here.platform:sdk-batch-bom_2.13:2.88.6')
}
  1. Add the dependencies that are declared in the BOM file without specifying version:
dependencies {
implementation platform('com.here.platform:sdk-batch-bom_2.13:2.88.6')

implementation "com.here.platform.data.processing:pipeline-runner_2.13"
}

After all steps are completed you can verify that all are configured correctly by running a command in a project root
directory:

./gradlew build

In case of success, the output looks like:

BUILD SUCCESSFUL in 2s

Pipelines runtime environment libraries

The HERE Workspace provides batch and streaming pipelines to run
location data-based applications. Each pipeline job is executed within a
specific runtime environment. This environment consists of the following:

  1. Java Runtime Environment (JRE)
  2. Apache Spark or
    Apache Flink framework

A batch (Spark) or streaming (Flink) environment comes with a list of packages
that are loaded by the Java Virtual Machine (JVM) and take precedence over any
other application-provided package.

To describe the list of packages that are part of the runtime environment, the
Data SDK provides two environment BOMs:

  • Scala 2.13:
    • environment-batch_2.13-4.0.3.pom
    • environment-stream-7.0.0.pom

Packages for a runtime environment are marked as provided in the
corresponding environment BOM. For example, the Apache Spark package
spark-core is marked as provided in environment-batch_2.13-5.0.0.pom:


<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_${scala.compat.version}</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
</dependency>

Notice that sdk-batch-bom_2.13 and sdk-stream-bom_2.13 files include the
corresponding environment-batch_2.13-5.0.0.pom and environment-stream-7.0.0.pom files.
A project that uses sdk-batch-bom_2.13 or sdk-stream-bom_2.13 does not need to explicitly
include the corresponding environment POM file. For more information about
the libraries provided by the pipeline runtime environments, see the corresponding
environment POM files.

Protobuf schemas

The HERE Workspace allows users to define their Protobuf schemas and distribute
them in the Workspace. The schemas are stored in a special repository which can be accessed only by means of
the HERE Maven Wagon plugin in Maven
projects or the
unofficial HERE SBT Resolver plugin
for SBT projects.

To start configuring and using the plugins, you
need HERE platform credentials.

Maven projects

To configure your project for use with the Maven Wagon plugin, follow the steps below:

  1. Include the Maven Wagon plugin in your project using the latest Maven version. For the version number, see
    the HERE Maven Wagon plugin page.
    <build>
      <extensions>
        <extension>
          <groupId>com.here.platform.artifact</groupId>
          <artifactId>artifact-wagon</artifactId>
          <version>${artifact.wagon.version}</version>
        </extension>
      </extensions>
    </build>
  1. Add the HERE Artifact Service reference into your project.
    The here+artifact-service://artifact-service placeholder will be replaced by the plugin dynamically based on your
    credentials.
    <repositories>
    
      <repository>
        <id>HERE_PLATFORM_ARTIFACT</id>
        <layout>default</layout>
        <url>here+artifact-service://artifact-service</url>
      </repository>
    </repositories>
  1. Add the dependency to the schema in your project, similar to
    the Administrative Place Profiles
    example below. Use the values for the schema you need to add.
    <dependencies>
      <dependency>
        <groupId>com.here.schema.rib</groupId>
        <artifactId>administrative-place-profiles_v2_java</artifactId>
        <version>2.88.0</version>
        <type>jar</type>
      </dependency>
    </dependencies>

To find the code snippet for your schema, go to
the HERE portal. Find a shared schema you want
to include and open the Artifacts tab.

For more information on the configuration of the Maven Wagon plugin,
see HERE platform Maven Wagon plugin.

SBT projects

To configure your project for use with the SBT Resolver plugin, follow the steps below:

  1. Register the sbt-resolver plugin by adding the following entry to the projects/plugins.sbt file:
    addSbtPlugin("com.here.platform.artifact" %% "sbt-resolver" % 2.0.39)
  1. Add the HERE Artifact Service reference into your project.
    The here+artifact-service://artifact-service placeholder will be replaced by the plugin dynamically based on your
    credentials:
    resolvers += "HERE_PLATFORM_ARTIFACT" at "here+artifact-service://artifact-service"
  1. Add the dependency to the schema in your project by including the following declaration in the build.sbt file,
    similar to
    the Administrative Place Profiles
    example below. Use the values for the schema you need to add.
    libraryDependencies += "com.here.schema.rib" %% "administrative-place-profiles_v2_scala"% "2.88.0"

To find the details of the schema, go to
the HERE portal. Find a shared schema you want
to include and open the Artifacts tab.

For more information on the configuration of the SBT Resolver plugin, see
the HERE SBT Resolver plugin
page.

Gradle projects

To configure your project for use with the Gradle Resolver plugin, follow the steps below:

  1. Include the Gradle Resolver plugin in your project using the latest Gradle version. For the version number, see
    the HERE Gradle Resolver plugin page.
       plugins {
           id 'com.here.platform.artifact.gradle' version '1.0.4'
       }
  1. Add the here() Artifact Service repository into your project.
      repositories {
         here()
       }
  1. Add the dependency to the schema in your project, similar to
    the Administrative Place Profiles
    example below. Use the values for the schema you need to add.
       dependencies {
           implementation "com.here.schema.rib:administrative-place-profiles_v2_java:2.88.0"
       }

To find the code snippet for your schema, go to
the HERE portal. Find a shared schema you want
to include and open the Artifacts tab.

For more information on the configuration of the Gradle Resolver plugin,
see HERE platform Gradle Resolver plugin.

For more information on how to create and publish a new schema,
see Create and Extend Schemas.