How to build a JAR File including the Data Client Library

This topic describes how to build a JAR file that includes the Data Client Library as well as all dependencies (fat-jar). The resulting JAR file can be used in either a pipeline or in your own application.

Note

The Data Client Library requires that each module/JAR have a reference.conf file, all of which are loaded individually. If you merge multiple JAR files into one JAR file, you also need to merge all your reference.conf files. If you do not, the defaults cannot be loaded and your project stops working.

If you use Maven to package your application, you can also use Apache Maven Shade Plugin support for Resource Transformers to merge all the reference.conf files on the build classpath into one file. sbt also offers similar plugins.

Use the following plugin configurations:

// sbt plugin
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "<latest-version>")

// sbt project
project.in(file("examples-project"))
  .settings(
    mainClass in assembly := Some("project.Main"),
    assemblyMergeStrategy in assembly := { defaultMergeStrategy },
  )

// merge strategy
val defaultMergeStrategy: String => MergeStrategy = {
  case x if Assembly.isConfigFile(x) =>
    MergeStrategy.concat
  case PathList(ps@_*) if Assembly.isReadme(ps.last) || Assembly.isLicenseFile(ps.last) =>
    MergeStrategy.rename
  case PathList("META-INF", xs@_*) =>
    xs map { _.toLowerCase } match {
      case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
        MergeStrategy.discard
      case ps@(x :: _) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") || ps.last.endsWith(".rsa") =>
        MergeStrategy.discard
      case "services" :: _ =>
        MergeStrategy.filterDistinctLines
      case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) | ("spring.tooling" :: Nil) =>
        MergeStrategy.filterDistinctLines
      case ("io.netty.versions.properties" :: Nil) =>
        MergeStrategy.first
      case _ :: "com.fasterxml.jackson.core" :: _ =>
        MergeStrategy.first
      case _ :: "commons-logging" :: _ =>
        MergeStrategy.first 
      case _ => MergeStrategy.deduplicate
    }
  case _ => MergeStrategy.first
}

<build>
    <finalName>my-project-name</finalName>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>1.5</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <shadedArtifactAttached>true</shadedArtifactAttached>
                        <shadedClassifierName>allinone</shadedClassifierName>
                        <artifactSet>
                            <includes>
                                <include>*:*</include>
                            </includes>
                        </artifactSet>
                        <transformers>
                            <transformer
                                    implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                <resource>reference.conf</resource>
                            </transformer>
                            <transformer
                                    implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <manifestEntries>
                                    <Main-Class>MyAwesomeMainClass</Main-Class>
                                </manifestEntries>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.6.1</version>
            <configuration>
                <source>17</source>
                <target>17</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>3.1.1</version>
            <configuration>
                <archive>
                    <manifest>
                        <mainClass>MyAwesomeMainClass</mainClass>
                    </manifest>
                </archive>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
            </configuration>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Note

When both spark-support and hadoop-fs-support are added to a project, it is possible that it may not work out of the box. Both of these modules implement HadoopFileSystem and bring in "org.apache.hadoop.fs.FileSystem" with different content. Depending on how the user is building the fat jar it may happen that one of these implementations gets discarded and thus filesystem implementation will not be found. In this case the merge strategy should be adapted that both implementation references are merged. Then run the following commands to build a fat-jar:

myMergeStrategy = {
  ...
  case PathList("META-INF", "services", "org.apache.hadoop.fs.FileSystem") =>
    MergeStrategy.concat
  ...
}

        ...
        <transformer
        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
            <resource>META-INF/services/org.apache.hadoop.fs.FileSystem</resource>
        </transformer>
        ...

You can build your fat jar by running this command:

sbt assembly

mvn package -Pplatform

Updated 21 days ago