The road so far….

November 1, 2012

Adapting in-compatible APIs

Filed under: java — Rahul Sharma @ 4:24 pm

In Apache Crunch we are supporting two versions of  Hadoop viz. 1.0.3 and 2.0-alpha. In such situations couple of times you could hit the issue of incompatible APIs. Hadoop version 1.0.3 contains couple of classes in the core-lib which are interfaces in version 2.0-alpha. Now if you need to implement an interface for one of the version how would you go about ? The APIs in the two versions are completely incompatible, so would you drop the idea and start supporting one version ? But that’s a big price to pay, thankfully there are couple of things to your rescue in such cases.

One way you could fix the issue is by using the munge plugin. In munge you have to implement the code in a commented if-else.  For example in our case I was implementing the TaskInputOutputContext class like :

class CustomContext{
TaskInputOutputContext getTestContext(){
TaskInputOutputContext context=null;
context=new TaskInputOutputContext(new Configuration(),new TaskAttemptID(),null,null,null){
 String getStatus(){return "newstatus";}

context=new TaskInputOutputContext(){........}
return context;

When you compile the code you have to enable  either  Hadoop_1.0.3 or Hadoop_2.0.2-alpha  variable.  The plugin sees that variable and enables the relevant blog of code.

But there are certain pitfalls of this solution. Most of the code is in comments so if you happen to do a format on the class then the complete thing goes for a toss. Also you have to make sure that the un-commented source which is generated at runtime gets compiled into a the final .class and not the original source.

Another way of addressing the original issue is via bytecode manipulation libs like Javassist. Yon can extend a class and do all the required changes in the following manner :

TaskInputOutputContext createTestContext() throws Exception {
   ProxyFactory factory = new ProxyFactory();
   Class superType = TaskInputOutputContext.class;
   Class[] types = new Class[0];
   Object[] args = new Object[0];
   if (superType.isInterface()) {
     factory.setInterfaces(new Class[] { superType });
   } else {
     types = new Class[] { Configuration.class, TaskAttemptID.class, RecordWriter.class, OutputCommitter.class, StatusReporter.class };
     args = new Object[] { conf, new TaskAttemptID(), null, null, null };
   factory.setFilter(new MethodFilter() {
     public boolean isHandled(Method m) {return m.getName().equals("getStatus");}
   MethodHandler handler = new MethodHandler() {
     public Object invoke(Object arg0, Method arg1, Method arg2, Object[] arg3) throws Throwable {return "newStatus";}
   return (TaskInputOutputContext) factory.create(types, args, handler);

There are some benefits of this way as compared to the previous solution. The code here is quite concise. I actually needed to implement only a few of the APIs so when I took the munge approach then I implement the complete classes but here as you can see there are method filters which work on method name.

This is a plus and a minus also, in the munge plugin the code that is un-commented gets compiled into the complete class. If  it happens that someone removed/renames the code in the implemented method then there is better chance at finding the issue. It will throw compile time errors, here using byte instrumentation it can be hard to find the issue as the code will  compile always eg rename the method to “getState”, you will not any error until runtime.

Besides these there are other wasy also like making seperate sub modules and then implementing the required but I feel its a quite costly step to take. But if you end up with a lot of things in such a state I think it looks like the step to take.

Anyhow there are solutions to your in-compatibility issues so lets start fixing them !

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: