The road so far….

August 29, 2014

Horizontal Scaling using Parallel file systems

Filed under: Linux — Rahul Sharma @ 8:45 am

If you are doing good business then you need to be prepared for scaling your services. Stateless services are easy to scale, but if there are state-full ones then you need to think sometime before you start trying to scale. Now we had one service that was building state on its local disc, like a invoice generation service where your customers can generate invoices for their customers in all kind of formats eg PDF, Excel etc. While it was easy to vertically scale the service, just shutdown the EC2 instance, change the type and start-up, but there is a ceiling to it. How can we scale such a service  horizontally without rewriting our component ?   (more…)

August 22, 2014

Limiting Fork Bomb in Docker

Filed under: Linux — Tags: , — Rahul Sharma @ 12:55 pm

Docker is a great tool for experimenting and learning. If you haven’t tried it till now then do give it try !  You can play with-in it without messing your own system. We can also built limitations for the container so that the host does not have any major impact in case the container goes berserk. While it was easy to put limitations for memory and CPU utilization, last few days I spend major amount of time in figuring out ‘how to handle a fork bomb ?’ (more…)

August 4, 2014

Introducing Hadoop Development Tools

Filed under: java — Tags: , — Rahul Sharma @ 12:54 pm

Few days back Apache Hadoop Development Tools a.k.a. HDT was released.  The projects aims at bringing plugins in eclipse to simplify development on Hadoop platform. This blog aims to provide an overview of few great features of HDT.

Single Endpoint :

The project can act as a single endpoint for your HDFS, Zookeeper and MR Cluster. You can connect to your HDFS/Zookeeper instance and browse or add more data. You can submit jobs to MR cluster and see status of all the running jobs.  (more…)