It’s Your Content, Claim The Logs For It

Blog @Bintray

Improved statistics we introduced last month give you much more information about the users that download open source software you publish than any other binaries distribution platform ever.

But why stop there? What if you want even more? One of the most powerful ways to slice and dice your download stats is to play with Apache-style logs. This standard, known as Common Log Format, is well familiar and there are tons of utilities for parsing, analyzing and generally going wild with these logs.

If you you’re an open source developer distributing your packages on Bintray, you may have wished for the logs. You may have also thought that since Bintray is SaaS and the platform’s logs are part of how the backend works, then you’ll need to settle with what we give you in the UI.

Well, wrong! The good news are – we give you all your…

View original post 49 more words

It’s Your Content, Claim The Logs For It

Blog @Bintray

Improved statistics we introduced last month give you much more information about the users that download open source software you publish than any other binaries distribution platform ever.

But why stop there? What if you want even more? One of the most powerful ways to slice and dice your download stats is to play with Apache-style logs. This standard, known as Common Log Format, is well familiar and there are tons of utilities for parsing, analyzing and generally going wild with these logs.

If you you’re an open source developer distributing your packages on Bintray, you may have wished for the logs. You may have also thought that since Bintray is SaaS and the platform’s logs are part of how the backend works, then you’ll need to settle with what we give you in the UI.

Well, wrong! The good news are – we give you all your…

View original post 49 more words

Be the First to Know. Really.

New and very sarcastic blog post 🙂

Blog @Bintray

So, you have an early-2000 style repository, like Maven Central:

And let’s say you are very, extremely interested to know when the new version of netty comes out.  We understand, it’s a natural addiction. How can you do it? Here are some ideas:

• Well, you can visit Maven Central every day. Couple of times a day.
• Oh, they have an RSS feed (did we mention early-2000?) It will notify you for everything being published to Maven Central. Well, probably you’ll spend all day reading those notifications, but eventually your eye might catch the project update you are looking for. You could also use Google Reader to filter the RSS stream for you… Oh, wait…
• Another idea! You can use their REST API. Just a simple query directly on their Solr will produce you a nice JSON file of 109 lines, and by a trivial parsing you’ll find the latest version…

View original post 225 more words

Bintray + GitHub = Synergistic Love Story

Blog @Bintray

First things first – Bintray is not a competitor of GitHub. They complete each other, not compete. Here’s how (I love vienn diagrams):

Bintray is an organic next step for developing software at GitHub – once your sources are built – distribute them from Bintray.
Our job is to make it as easy as possible for you, our fellow GitHubber. Here’s what you get:

First, sign up to Bintray using GitHub:

Authorize Bintray for GitHub, fill the needed details, and you’re done.

Naturally, login using GitHub too:

Next step is the only one you’ll have to do manually, without GitHub integration – creation of a repository¹. Don’t forget to select the right type!

Once that done, we are back to GitHub integration again. Just click on Import From Git:

Get your stuff from to Bintray in two simple steps:

1. Select the desired GitHub repositories to become Bintray packages

View original post 232 more words

On Using the Right Tool for the Job

Disclaimer: The code samples in the post below are for brain teasing only. Do not, I repeat, do not ever do such things for any other purpose!

A friend of mine is a big fan of puzzlers. Any puzzlers, including programming ones. Here’s his latest and greatest:

Write some code in the static initializer to make the assert pass:

public class Test {
 static {
//Write some code here
 }

public static void main(String[] args) {
 Integer a = 20;
 Integer b = 20;
 assert (a + b == 60);
 }
}

Don’t forget to enable assertions if you want to solve this one (-ea as VM parameter).

If you solved it, or don’t feel like puzzlers, read on for the solution and the punchline.

QCon 2012 – Perfect as Everything in London Should Be

This post was originally posted on ‘From the Frog’s Mouth‘.
You’re welcome to comment here or there.

It was JFrog‘s second QCon London, and it just gets better. Imagine: even the London weather was perfect, not to mention the sessions, booth traffic, show organization and food (what, you say, good English food? Well, great IndoPak food, at least). Due to high demand by sponsors, the exhibition took place on two floors (as opposed to one floor last year). Our JFrog booth was located in the same place as in 2011. We’re getting used to the place and are looking forward to returning next March!

The speaker panel was extremely impressive, featuring gurus like Martin Fowler, Adrian Cockcroft (the man behind Netflix’ cloud), Dwight Merriman (co-creator of MongoDB), Damien Katz (creator of CouchDB) and Rich Hickey (creator of Clojure).

The conference started with two training days – six full-day tutorials each. From my perspective, the most interesting two tutorials were “Cloud Architecture” by Adrian Cockcroft, where he shared the architecture, best practices and decisions behind Netflix’ cloud (which Artifactory is proud to be a part of) and “Continuous Delivery” by ThoughtWorks’ Tom Sulston (that’s as close to the roots of the famous “Continuous Delivery” book as you can get). For me, the most fascinating thing in the Continuous Delivery process as ThoughtWorks sees it, is that its virtues are exactly the same as we based our Artifactory upon back in 2006: DevOps automation and rapid release cycle. We appreciated the validation of our concept.

The remaining three days of the week were all-day sessions. It’s impossible to review such a significant number of talks in one blog post, so I’ll concentrate only on the excellent keynote addresses (with one exception).

Martin’s conference-opening keynote speech was about data. The main feature of modern data is that it is bigger than you think. Polyglot storage in general and various kinds of NoSQL look like the right solution.

My favorite keynotes are usually the evening ones. A beer in your hand makes any amusing talk even more enjoyable. One named “Developers Have a Mental Disorder” I couldn’t miss! Greg Young gave a great show, funny and entertaining, about serious dilemmas in software development that we, the developers, prefer to ignore. The brightest example, of course is the downside of DRY (did you ever think about one?). By removing duplication, we increase coupling, which can be much worse.

Thursday morning’s keynote address was delivered by Rich Hickey. He spoke about the differences between “Simple” and “Easy”. Sounds pretty similar, but in fact they are very far from being the same. Antonyms to the rescue: simple vs. complex, while easy vs. hard. Now it’s clear – we need to strive to prevent and remove complexity (go simple) without being afraid of the hard. Choosing easiness may end up creating complexity. Things which are easy, but complex, include OOP, mutable state, imperative loops and frameworks (especially ORM). Things which are simple, but not necessarily easy (at least not until you get them), are LISP-ish languages, like Clojure.

My session also took place on Thursday, in relatively small room, about 70 people. I was more than happy to see that it was almost packed. I spoke about building trust in your build process by selecting the right tools for the job (of course, we consider Artifactory as one). I also spoke about the problems of DevOps in the word of Continuous Delivery in the cloud and the rapid release cycle of SaaS applications. I stressed the huge timeshare of binaries in ALM process and the importance of using a tool that really understands binaries to deal with them. That’s exactly the reason why we developed Artifactory.

Half of my session was dedicated to live demos, which went smoothly, incredible as it may sound. According to the feedback received, my talk was well accepted, and hopefully will be useful to some of the attendants for building easier and more reliable release processes. The Q&A session continued at our booth, where we repeatedly did live demonstrations and received excellent feedback each time. If you want to get a feeling of my talk, here’s the slide-deck.

Friday was the last day of the conference. It started hard with a highly technical keynote address by John Allspaw with a scary name: “Resilient Response In Complex Systems”. For someone like me, who doesn’t deal on a daily basis with Disaster Recovery, the session was astonishing. Looking behind the curtain of that kitchen reveals a totally different way of thinking and planning. It may be how individuals and teams have to perform during a disaster (e.g. personal heroism is bad even if successful; it sends the wrong message to the team), or simulating disasters on live production systems (I never could even dare to think about that). The most obvious, but still eye-opening advice that John gave is to learn from successes, not only from failures. It can give us a lot of information and happens much more frequently, no? The only organization with which I am familiar that embraces that technique is the Israeli Air Force.

To sum up, the conference was great by every measure: technical sessions, attendance, networking, Artifactory exposure, and after-show quality time. Thank you, InfoQ, for this wonderful event in London. QCon was a great starting shot for JFrog’s “Busy March”. You still can catch Fredric and Yoav giving talks on various events in US and Europe.

Dependency Management with .NET – Doing it Right

This post was originally posted on ‘From the Frog’s Mouth‘.
You’re welcome to comment here or there.

The problem of dependency management is neither new nor original, it exists in all development platforms, and .NET is no different.

Let’s go through different solutions and see how they perform. I’ll list them here in no particular order.

Unified (as much as possible) Logging Using SLF4J

Integrating, integrating, integrating. That’s what we do in Java enterprise development. Persisting objects with Hibernate wrapped by JPA using C3Po (or JTA?) (or MongoDB over Morphia?), processed with JBMP, created by JAXB (jackson-json?) from JAX-RS scheduled by Quartz … (a few dozen frameworks later) … all this glued with Spring (or Guice?) deployed on Jetty (or Tomcat, JBoss, Resin?) into cluster by Terracotta (or Hadoop, GigaSpaces, JBoss cache, Infinispam?). Ah, and all this built using Maven Gradle with Artifactory on Jenkins. I sure forgot ½ of the frameworks we constantly use.

Generally we don’t mind much about the internals of the frameworks we use (as long as they are good) – the whole encapsulation stuff is the last undoubted good thing. But except for the API (part of which is the configuration) frameworks have another user-facing end – the logging. When we build a system we want it to behave as one system – single configuration from one end, and single log from another (break it to different files, if you wish, but it should still be a unified logging system).

The reality is that there is no standard de-facto for logging. The standard de-jure – JUL, is not very popular because of its lack of functionality (compared to alternatives) and its suboptimal performance. And then there is Log4J, which almost became standard, but did not. And there is logback, which is a Log4J trashover, and there are facades (JCL and SLF4J), which try to unite all this zoo, and some others, which you have probably never heard of, like syslog4j*, logging framework by the Object Guy, jLo, MonoLog, Lumberjack, Houston, JTraceDump, qflog, LN2, TracingClassLoader, SMTPHandler, Log4Ant, Simple Log, Log Bridge, Craftsman Spy, Pencil, JDLabAgent, Trace Log, JDBC Logger, LimpidLog and Microlog.

Let it be, you’d say – why not have many logging tools, which are good and diverse! Well, the problem, as I’ve already mentioned, is that they leak out of the frameworks. Their diverse configuration leaks from one end, while their diverse output from another. Spring uses Log4J over JCL. So does Hibernate. Jetty uses Logback over SLF4J. Some (like Terracotta modules) use plain Log4J, Jersey uses JUL.  This means we end up with 5 separate configurations (Log4J, SLF4J, Logback, JCL and JUL) and 3 different types of log files (Log4j, Logback and JUL). What a system!

To make the long story short – How can we achieve the desired consolidation? Clearly, we need a facade. There are two most commonly used – SLF4J and JCL. JCL is known for its classloader hell, SLF4J is newer, better performing, smarter, simplier to use and generally provides better quality for the same buck (well, no buck – both are open source, of course), so we’ll stick to it. SLF4J is an adapter – thin layer of API to and from different logging implementations. Yap, both ways. It means with SLF4J we can use JUL API on top and log using Log4J in the bottom!

First we need to pick an actual logger. Log4j was considered the best choice up until recently (2006) when Ceki Gülcü decided he needed a fresh start and rewrote from scratch a new Java logging framework, just better than log4j, called Logback. We can give it a try as our underlying logging implementation (we can switch in a moment, as we are using  good facade, remember?).

So, here’s what we have to do:

1. Establish our own good logging:
   1. Add Logback to our classpath
   2. Add SLF4J API to our classpath

   Done here. Now our own brand new code will use top-notch logging.

2. Now for the tricky part. Let’s make the example stack I listed above taking configuration from one source (our config files) and writing to one target (files, listed in our configuration)
1. All the tools using SLF4J will just work. That includes dozen of Apache projects, inc. Camel and Mina, some SpringSource projects and many others.
2. Now let’s start rolling with all the rest. This is how you do it (click to enlarge):
   
1. Jakarta Commons Logging:
1. Remove commons-logging.jar from your classpath. Usually, it is transitive dependency from the framework, so you need to instruct your build tool on how to do it. What a lucky coincidence, I just wrote short and instructive blog post about how to do it!
2. Add jcl-over-slf4j.jar instead. It contains alternative commons-logging API implementation, so the code will run just fine.
2. Log4J:
1. Same goes here! Remove log4j.jar from your classpath (Again, it would usually be a transitive dependency from the framework, look here).
2. Add log4j-over-slf4j.jar instead. It contains alternative log4j API implementation, so the code will run just fine.
3. JUL:
1. Well, you can’t remove JUL from classpath (it’s a part of the JRE, dude). For the same reason SLF4J can’t reimplement JUL’s API.
2. Add jul-to-slf4j.jar. It will translate java.util.logging.LogRecord objects into their SLF4J equivalent.
3. Install SLF4JBridgeHandler and LevelChangePropagator.
4. Expect 20% decrease in performance (so use it wisely).

All done. Now both our code and all the 3rd paries configured from single source and write to single target. Hooray!

* syslog4j claims it is cross-platform. Well,  I’ll just quote: “Is Syslog4j cross-platform? Yes! Syslog4j UDP/IP and TCP/IP clients should work in any typical Java JRE environment.”

Banning Transitive Dependencies With Maven2/3, Gradle and Ivy

Oh, you are using build tool with dependency management? Good! Be it Maven2/3, Gradle or Ivy, your life as devops or developer is much easier. Until you hit it. The evil transitive dependency. How can it be evil you ask? When the classes in it clash with the classes you really need.  Here’s some use-cases:

1. Same dependency, different jar names, two examples here:
1. The Jakarta Commons renaming effort: commons-io:commons-io:1.3.2 and org.apache.commons:common-io:1.3.2
2. The Spring Framework artifacts naming convention alternatives: spring-beans, spring-context, etc in repo1 versus org.springframework.beans, org.springframework.context, etcin
   SpringSource EBR.
2. Different packaging of the sample classes, many examples here:
1. OSGi repackagings: asm:asm:3.2 and org.objectweb.asm:com.springsource.org.objectweb.asm:3.2.0
2. Modularization of Spring 2.5.6: as single jar and as spring-whatever multiple modules
3. Xerces and Xalan are included in JDK since 1.5. They are still present as transitive dependencies in all the tools which support JDK 1.4.
4. Alternative packagings with and without dependencies: cglib:cglib and cglib:cglib-nodep
5. Project merges like Google collections, which are now included in Google Guava
3. Deliberately reimplemented interfaces, for example for bridging legacy APIs to new implementation, such as in SLF4J.
4. Your patches for 3rd-party tools.

All those may end up with 2 or more classes with the same name in the classpath. Why it is bad? Java class identifier consists of fully-qualified class name and the classloader that loaded it, so if two classes with the same name reside in same classpath JVM considers them to be the same class, and only one of them will be loaded. Which one? The first classloader encounters. Which one will it be? You have no idea.
When the duplicated classes are exactly the same, you will never notice. But if the classes are different, you’ll start getting runtime exceptions, such as NoSuchMethodError, NoClassDefFoundError and friends. That’s because other classes expect for find one API, but encounter another one – wrong class was loaded first. Not fun.

Now, when you know how evil they are, let’s take those bastards down!

Maven 2/3

There is no simple way (Maven’s tagline) to exclude some dependency from all the scopes. I’ll show two cases – manual exclusion and working with IntelliJ IDEA:

1. Stage 1: exclude all the banned dependencies one by one:
1. Manually edit Maven’s poms
1. For each evil dependency:
2. Find which top-level dependency brings the evil transitive hitcher with it. This is done by using Maven Dependency Plugin:

   mvn dependency:tree -Dincludes=commons-logging:commons-logging

3. You’ll get something like this:

   [INFO] com.mycompany.myproduct:rest-client:1.0
   [INFO] \- org.springframework:spring-webmvc:jar:3.0.5.RELEASE:compile
   [INFO]    \- org.springframework:spring-core:jar:3.0.5.RELEASE:compile
   [INFO]       \- commons-logging:commons-logging:jar:1.1.1:compile

4. Go to the pom.xml with your dependency management (you use dependency management, don’t you? If you don’t, don’t tell anyone, go and start using it) find spring-webmvc dependency and add an exclusion to it:

   1     <dependency>
   2     	<groupId>org.springframework</groupId>
   3     	<artifactId>spring-webmvc</artifactId>
   4     	<version>3.0.5.RELEASE</version>
   5         <exclusions>
   6             <exclusion>
   7                 <artifactId>commons-logging</artifactId>
   8                 <groupId>commons-logging</groupId>
   9             </exclusion>
   10         </exclusions>
   11     </dependency>

2. Working with IntelliJ IDEA:
   
1. Open Maven Dependencies Graph.
2. Filter it by the dependency you are looking for.
3. Select it and press Shift-Delete.
2. Good job! Your nailed them down in the current version of your build. But what happens when someone adds a new 3rd party dependency and brings some bad stuff with it as transitives? You need to protect your build from this scenario. So, stage 2: Fail the build if one of the banned dependencies ever added to the build with Maven Enforcer Plugin. Add the plugin to your root project pom:

   1 <project>
   2   <build>
   3     <plugins>
   4       <plugin>
   5         <groupId>org.apache.maven.plugins</groupId>
   6         <artifactId>maven-enforcer-plugin</artifactId>
   7         <version>1.0</version>
   8         <executions>
   9           <execution>
   10             <id>enforce-banned-dependencies</id>
   11             <goals>
   12               <goal>enforce</goal>
   13             </goals>
   14             <configuration>
   15               <rules>
   16                 <bannedDependencies>
   17                   <excludes>
   18                     <exclude>commons-logging</exclude>
   19                     <exclude>cglib:cglib</exclude>
   20                   </excludes>
   21                 </bannedDependencies>
   23               </rules>
   24               <fail>true</fail>
   25             </configuration>
   26           </execution>
   27         </executions>
   28       </plugin>
   29     </plugins>
   30 </build>
   31 </project>

3. As I mentioned, using the Enforcer plugin won’t exclude the unwanted dependencies, it only will fail the build. Once that happened (and trust me, it will), you need to go and exclude them manually, as described in Stage 1 above.

And we are done with Maven. Not fun? Switch your build tool!

Ivy

Well, comparing to Maven it’s emabrassing how easy is to add global exclusion in Ivy. All you need to do is add exclude tag, and it will do the job for all the transitive dependencies, both in current and future use:

1 <dependencies>
2     <dependency org="org.springframework" name="spring-webmvc"
3 rev="3.0.5.RELEASE" conf="compile->default"/>
4     <exclude org="commons-logging"/>
5 </dependencies>

Done.

Gradle

Since Gradle uses Ivy under the hood, here comes the same ease, but even groovier:

1     configurations {
2         all*.exclude module: 'commons-logging'
3         all*.exclude group: 'cglib', module: 'cglib-nodep'
4     }

That’s all! Now your code is bullet-proof from classloading conflicts and you can do nasty class-replacing stuff, for logging or pleasure.

PAX 2010

Just back from Project Automation Experience 2010.

Guess what? It was awesome!

Here’s my summary:

General:

The conference was combined with Rich Web Experience 2010, which probably, was the right thing to do for the first ever project automation seminar. It was announced 2 months ago, bringing 52 registrants (out of ~400 total for both events). Pretty impressive for such a short notice on such a narrow subject. The organization was fantastic, everything felt well-planned and well-orchestrated. It was my second NFJS conference, and like with the first one, they delivered!

Sessions:

The first speaker in the conference was the one and only Douglas Crockford in his notorious Quality talk. Top quality (pun intended) – funny, entertaining, and touching the right points. Bottom line – read The Mythical Man-Month and Literate Programming.

Yet another keynote was devoted to actually Project Automation. Hans Dockter laid down his vision on the topic, the idea being – we are entering very interesting times, better understanding of the needs combining with the right tools will enable us whole new level of project automation, way beyond what we are used to today.

Lots of fun and enriching talks, like Tim Berglund‘s Complexity Theory and Software Development or an experts panel, moderated by Ted Neward (who honestly tried to recall who I was and where did we met), and tons of useful and deep sessions and workshops, just to name a few: Fred‘s modularity and smart BRMs talks, Kohsuke’s updated Doing More with Hudson, Git, Sonar and Liquibase sessions from Matthew McCullough, Olivier Gaudin and Tim Berglund. The integration with Rich Web Experience gave the participans the opportunity to mix and match the sessions, and there was a lot of stuff to attend – HTML5, CSS3, Flash, iOS and Android development, Grails, Wicket and what’s not! 10 parallel sessions in any given time, now go choose one!

The sessions I gave:

First of all, it was my first speaking experience outside of Israel. I think I did well. At least the feedbacks say so 🙂 The high speaker:attendee ratio produced small classes with live interaction, I loved it. As I suspected, 90 minutes are too long, looks like 60 minutes is my favorite format, but I managed to keep the audience awake 🙂

Here are my sessions:

• Maven 2 Gradle how to move away from Maven2 to Gradle (using gradle2maven, maven-metadata plugin or even manually). Demo included converting the whole Artifactory project to Gradle.
• Java Build Tools Jungle – overview the tons of Java build tools. Winners are (in that order): Gralde, Buildr, SBT.

By the way, Prezi rocks, as usual.

My next goal – speaking in a bigger seminar around spring 2011, probably on different topic, more relevant to my new job.

Florida:

December 1st – 30ºC, sun and ocean. That’s a typical room view. No additional comments needed, I guess.

Everglades and alligators rock too. It looks like this.

Olivier Gaudin