GuidesChangelogData Inspector Library API Reference
Guides
  1. Data Processing Library
  2. Get started with Data Processing Library

Data Processing Library Java bindings

Data Processing Library Java bindings

The Data Processing Library provides a Java interface to support compilers written entirely in Java. This interface exposes most of the functionality available in Scala with some exceptions.

Java specific classes are defined in the com.here.platform.data.processing.java package and its sub-packages.

The Data Processing Library Java classes and methods have identical names to their Scala counterparts. However, method signatures may differ as Scala specific types are replaced by Java native types. The following sections describe these differences in detail.

Type aliases

In Java, types aliases are not used. For instance, in the Scala API the InKey and OutKey aliases of type Partition.Key are used when input keys and output keys are expected, respectively. However, the Java API uses a single type Key, instead.

Type aliases in Scala:

def compileInFn(in: (InKey, InMeta)): Iterable[(OutKey, IntermediateData)] = {

  // InKey and OutKey are aliases of Key
  // InMeta is alias of Meta
  // ...

Equivalent code in Java:

public Iterable<Pair<Key, IntermediateData>> compileInFn(Pair<Key, Meta> in) {
  // ...

The Symbol class

In many Scala classes, the Symbol class is used for methods that represent IDs or names. However, the Java counterpart uses the String class instead.

Symbol class in Scala:

// Catalog.Id and Layer.Id are aliases of Symbol
val outKey = inKey.copy(catalog = Default.OutCatalogId, layer = Layer.Id("output-layer"))

Equivalent code in Java:

// Default.OutCatalogId() and the layer ID are String objects
Key outKey = inKey.withCatalog(Default.OutCatalogId())
                  .withLayer("output-layer");

Scala collections

Scala collections are replaced by their Java counterparts.

Collections in Scala:

def resolveFn(src: (InKey, InMeta)): Map[RefName, Set[InKey]] = {
  // ...

Equivalent code in Java:

public Map<String, Set<Key>> resolveFn(Pair<Key, Meta> src) {
  // ...

Tuples and options

Tuples of 2 elements are replaced by the com.here.platform.data.processing.java.Pair class in the Java API. The scala.Option type is replaced by java.util.Optional.

Option type in Scala:

def compileOutFn(outKey: OutKey, intermediate: Iterable[IntermediateData]): Option[Payload] = {

  if (intermediate.isEmpty) {
    None
  } else {
    Some(Payload(encodeData(intermediate)))
  }
}

Equivalent code in Java:

public Optional<Payload> compileOutFn(Key outKey, Iterable<IntermediateData> intermediate) {

  if (intermediate.iterator().hasNext()) {
    return Optional.of(new Payload(encodeData(intermediate)));
  } else {
    return Optional.empty();
  }
}

Scala mixins

Mixins in the com.here.platform.data.processing.compiler package, should be combined together to implement a compiler. The Java API represents these mixins as interfaces. All traits that must be mixed-in in Scala are interfaces that you implement in Java.

Compiler mixins in Scala:

class MyCompiler
  extends RefTreeCompiler[IntermediateData]
  with CompileInFnWithRefs[IntermediateData]
  with CompileOut1To1Fn[IntermediateData] {

Equivalent code in Java:

class MyCompiler
    implements RefTreeCompiler<IntermediateData>,
        CompileInFnWithRefs<IntermediateData>,
        CompileOut1To1Fn<IntermediateData> {

Some Java classes provide the same functionality of multiple Scala traits mixed in together.

For instance, extending a PipelineRunner in Scala:

object Main extends PipelineRunner with DriverSetupWithBuilder {

  // configureCompiler provided by DriverSetupWithBuilder
  def configureCompiler(completeConfig: CompleteConfig,
                        context: DriverContext,
                        builder: DriverBuilder): builder.type = {
    // ...

Equivalent code in Java:

class Main extends PipelineRunner {

  // configureCompiler directly provided by PipelineRunner
  @Override
  public DriverBuilder configureCompiler(
      CompleteConfig completeConfig, DriverContext context, DriverBuilder builder) {
    // ...

Scala classes without Java specific interface

A few Scala classes do not have Java counterparts, and can be used directly from Java. These include:

  • Classes in the com.here.platform.pipeline.logging package, except the ContextAwareContextLogger class, of which a Scala version (com.here.platform.pipeline.logging.scaladsl.ContextAwareLogger) and a Java version (com.here.platform.pipeline.logging.javadsl.ContextAwareLogger) exist.
  • com.here.platform.data.processing.statistics.Statistic
  • com.here.platform.data.processing.driver.job.ToProcess and subclasses

Some of these Scala classes are meant to be used in match expressions, a feature that is not available in Java.

Pattern matching in Scala:

val processingType = driverContext.jobVersions.inCatalogs(id)

val version = processingType match {
  case NoChanges(v)  => v
  case Changes(_, v) => v
  case Reprocess(v)  => v
}

is replaced by a sequence of isInstanceOf predicates in Java:

ProcessingType processingType = driverContext.jobVersions().inCatalogs().get(id);

long version;

if (processingType instanceof NoChanges) {
  version = ((NoChanges) processingType).version();
} else if (processingType instanceof Changes) {
  version = ((Changes) processingType).version();
} else if (processingType instanceof Reprocess) {
  version = ((Reprocess) processingType).version();
} else {
  throw new RuntimeException("Match error");
}

Currying

Some classes provide their functionality through curried methods, such as the LogContext class' method withChild. In the Java version, the body part of the method's call is an explicit lambda expression.

So the following Scala code:

val logger = new ContextAwareLogger(getClass) // or use the one provided by ContextLogging
LogContext.withChild("my-context-id", id) {
  val payload = retriever.getPayload(in.key, in.meta)
  logger.debug(s"Retrieved ${payload.length} bytes")
}

becomes:

ContextAwareLogger logger = new ContextAwareLogger(getClass());
LogContext.withChild(
    "my-context-id",
    id,
    () -> {
      Payload payload = retriever.getPayload(in.getKey(), in.getValue());
      logger.debug("Retrieved {} bytes", payload.length());
    });

Implicit arguments

Implicit arguments are not supported in Java, thus they are appended to the argument list and you must provide them explicitly.

This includes Scala ClassTags used in some methods to carry along all type information available at compile time, to runtime.

def buildTask(driverBuilder: DriverBuilder,
              compiler: Direct1ToNCompiler[IntermediateData]): DriverTask = {

  val taskBuilder = driverBuilder.newTaskBuilder("task1")
  taskBuilder.withDirect1ToNCompiler(compiler).build()
}

In the Java counterpart, a class object needs to be explicitly passed to withDirect1ToNCompiler:

DriverTask buildTask(
    DriverBuilder driverBuilder, Direct1ToNCompiler<IntermediateData> compiler) {

  TaskBuilder taskBuilder = driverBuilder.newTaskBuilder("task1");
  return taskBuilder.withDirect1ToNCompiler(compiler, IntermediateData.class).build();
}

For a complete list of features currently supported in Java, refer to the API Reference.

Updated 21 days ago