null

NullPointerException, probably…

Banning Transitive Dependencies With Maven2/3, Gradle and Ivy

with 13 comments

Oh, you are using build tool with dependency management? Good! Be it Maven2/3, Gradle or Ivy, your life as devops or developer is much easier. Until you hit it. The evil transitive dependency. How can it be evil you ask? When the classes in it clash with the classes you really need.  Here’s some use-cases:

  1. Same dependency, different jar names, two examples here:
    1. The Jakarta Commons renaming effort: commons-io:commons-io:1.3.2 and org.apache.commons:common-io:1.3.2
    2. The Spring Framework artifacts naming convention alternatives: spring-beans, spring-context, etc in repo1 versus org.springframework.beans, org.springframework.context, etcin
      SpringSource EBR.
  2. Different packaging of the sample classes, many examples here:
    1. OSGi repackagings: asm:asm:3.2 and org.objectweb.asm:com.springsource.org.objectweb.asm:3.2.0
    2. Modularization of Spring 2.5.6: as single jar and as spring-whatever multiple modules
    3. Xerces and Xalan are included in JDK since 1.5. They are still present as transitive dependencies in all the tools which support JDK 1.4.
    4. Alternative packagings with and without dependencies: cglib:cglib and cglib:cglib-nodep
    5. Project merges like Google collections, which are now included in Google Guava
  3. Deliberately reimplemented interfaces, for example for bridging legacy APIs to new implementation, such as in SLF4J.
  4. Your patches for 3rd-party tools.

All those may end up with 2 or more classes with the same name in the classpath. Why it is bad? Java class identifier consists of fully-qualified class name and the classloader that loaded it, so if two classes with the same name reside in same classpath JVM considers them to be the same class, and only one of them will be loaded. Which one? The first classloader encounters. Which one will it be? You have no idea.
When the duplicated classes are exactly the same, you will never notice. But if the classes are different, you’ll start getting runtime exceptions, such as NoSuchMethodError, NoClassDefFoundError and friends. That’s because other classes expect for find one API, but encounter another one – wrong class was loaded first. Not fun.

Now, when you know how evil they are, let’s take those bastards down!

Maven 2/3

There is no simple way (Maven’s tagline) to exclude some dependency from all the scopes. I’ll show two cases – manual exclusion and working with IntelliJ IDEA:

    1. Stage 1: exclude all the banned dependencies one by one:
      1. Manually edit Maven’s poms
        1. For each evil dependency:
        2. Find which top-level dependency brings the evil transitive hitcher with it. This is done by using Maven Dependency Plugin:
          mvn dependency:tree -Dincludes=commons-logging:commons-logging
        3. You’ll get something like this:
          [INFO] com.mycompany.myproduct:rest-client:1.0
          [INFO] \- org.springframework:spring-webmvc:jar:3.0.5.RELEASE:compile
          [INFO]    \- org.springframework:spring-core:jar:3.0.5.RELEASE:compile
          [INFO]       \- commons-logging:commons-logging:jar:1.1.1:compile
        4. Go to the pom.xml with your dependency management (you use dependency management, don’t you? If you don’t, don’t tell anyone, go and start using it) find spring-webmvc dependency and add an exclusion to it:
          1     <dependency>
          2     	<groupId>org.springframework</groupId>
          3     	<artifactId>spring-webmvc</artifactId>
          4     	<version>3.0.5.RELEASE</version>
          5         <exclusions>
          6             <exclusion>
          7                 <artifactId>commons-logging</artifactId>
          8                 <groupId>commons-logging</groupId>
          9             </exclusion>
          10         </exclusions>
          11     </dependency>
      2. Working with IntelliJ IDEA:
        IntelliJ IDEA Maven Dependencies
          1. Open Maven Dependencies Graph.
          2. Filter it by the dependency you are looking for.
          3. Select it and press Shift-Delete.
    2. Good job! Your nailed them down in the current version of your build. But what happens when someone adds a new 3rd party dependency and brings some bad stuff with it as transitives? You need to protect your build from this scenario. So, stage 2: Fail the build if one of the banned dependencies ever added to the build with Maven Enforcer Plugin. Add the plugin to your root project pom:
      1 <project>
      2   <build>
      3     <plugins>
      4       <plugin>
      5         <groupId>org.apache.maven.plugins</groupId>
      6         <artifactId>maven-enforcer-plugin</artifactId>
      7         <version>1.0</version>
      8         <executions>
      9           <execution>
      10             <id>enforce-banned-dependencies</id>
      11             <goals>
      12               <goal>enforce</goal>
      13             </goals>
      14             <configuration>
      15               <rules>
      16                 <bannedDependencies>
      17                   <excludes>
      18                     <exclude>commons-logging</exclude>
      19                     <exclude>cglib:cglib</exclude>
      20                   </excludes>
      21                 </bannedDependencies>
      23               </rules>
      24               <fail>true</fail>
      25             </configuration>
      26           </execution>
      27         </executions>
      28       </plugin>
      29     </plugins>
      30 </build>
      31 </project>
    3. As I mentioned, using the Enforcer plugin won’t exclude the unwanted dependencies, it only will fail the build. Once that happened (and trust me, it will), you need to go and exclude them manually, as described in Stage 1 above.

And we are done with Maven. Not fun? Switch your build tool!

Ivy

Well, comparing to Maven it’s emabrassing how easy is to add global exclusion in Ivy. All you need to do is add exclude tag, and it will do the job for all the transitive dependencies, both in current and future use:

1 <dependencies>
2     <dependency org="org.springframework" name="spring-webmvc"
3 rev="3.0.5.RELEASE" conf="compile->default"/>
4     <exclude org="commons-logging"/>
5 </dependencies>

Done.

Gradle

Since Gradle uses Ivy under the hood, here comes the same ease, but even groovier:

1     configurations {
2         all*.exclude module: 'commons-logging'
3         all*.exclude group: 'cglib', module: 'cglib-nodep'
4     }

That’s all! Now your code is bullet-proof from classloading conflicts and you can do nasty class-replacing stuff, for logging or pleasure.

Written by JBaruch

22/06/2011 at 08:39

Posted in Build

Tagged with , , ,

13 Responses

Subscribe to comments with RSS.

  1. […] your build tool on how to do it. What a lucky coincidence, I just wrote short and instructive blog post about how to do […]

  2. There is another way with Maven, though it’s hacky – add the dependency you want to exclude as a top level dependency with a scope of provided. Maven will then leave it out of any runtime assemblage.

    The good thing about this is that it is global, so you don’t run the risk of the dependency being reintroduced later. The (very) bad thing is that Maven will make that library available in its entirety at compile time, so there is a possibility of compiling code against classes/methods that won’t be there at runtime. Still, it’s a possibility.

    (Baffling that Maven haven’t made global exclusions a priority – here’s the ticket from 2006 http://jira.codehaus.org/browse/MNG-1977 with 118 votes and 85 watchers.)

    Rob

    22/06/2011 at 17:33

    • Hmm, yeah, right… You could do that too. But I’d still advocate for replacing the build tool🙂

      jbaruch

      23/06/2011 at 06:38

  3. These are design and architecture issues that should be addressed at that level. These are the sorts of libraries that are heavily re-used throughout an org so it is worth the effort to make an enterprise tech stack available as part of a common architectural approach.

    Namespace conflict can also be resolved quite simply by re-ordering your dependencies, they get put on the classpath in the order you list them so you can explicitly put known conflicts at the end. Handy if there is only some package conflict but you still need other packages.

    Ivy and Gradle are nice alright but there is no way to standardize a build process across an organization with them, so I’ll stick with Maven where the overall comprehension is much higher. In your case, you wont see *why* it was excluded or which items were bringing it in. So while Maven is *embarassingly* harder it is better documented and easier to maintain over the long haul.

    onCommit

    24/06/2011 at 15:54

    • Re-ordering dependencies is very weak way to resolve the conflicts. The dependencies are used in many modules and building on tags order in all the XML files and keeing that order when dependencies added/deleted to prevent fatal exception? Very unstable.

      Considering Ivy and Gradle not being able to standardize a build process across an organization – what a strange point to make! Why would you thinks so?

      jbaruch

      26/06/2011 at 06:02

  4. […] over the years I have become increasingly disappointed with the quality of the provided software (as in the examples here). It is also emerging in OSGi applications to have this full dependency resolution, however, there […]

  5. wow, this helped me solve the problem. Thank you! I searched all over internet frantically for almost a day.

    nikulkarni

    03/03/2013 at 07:18

    • Glad you liked it! Feel free to share (twitter, facebook, linkedin) to save the day of searching for the next guy🙂

      jbaruch

      03/03/2013 at 08:49

  6. Wow, a post advocating stepping back in time..

    Do you remember what a nightmare it was when we didn’t have transitive dependencies under Ant/Maven 1?

    I’d much prefer the suggestion from Evgeny Goldin above – find the problems early, but don’t throw the baby out with the bathwater.

    Simon ML

    09/04/2013 at 11:29

    • Not sure I follow you. Are you suggesting that I am advocating giving up on transitive dependencies? If so, probably my writing isn’t so good. I never intended to write such a thing.

      jbaruch

      09/04/2013 at 11:40

  7. Wow that bug https://issues.apache.org/jira/browse/MNG-1977 has an update – after 10 years – “Starting with Maven 3.4, you can manage dependencies to optional in dependency management. That is the same as excluding them globally when not explicitly declared as a direct dependency.”

    erandall

    21/06/2016 at 19:11


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: