In this blog post, I will describe two memory-related issues that we have recently experienced on our 190-node Apache Hadoop cluster at Spotify.

We have noticed that some nodes were suddenly marked dead by both NameNode and JobTracker. Although we could ping them, we were unable to ssh into them, what often suggests some really heavy load on these machines. When looking at Ganglia graphs, we have discovered that all nodes that were marked dead have one common issue – a heavy swapping (in case of Apache Hadoop, the practice shows that a heavy swapping of JVM process usually means some kind of unresponsiveness and/or even the “death”).

The slides from the quick presentation that I gave at Spotify as part of “knowledge sharing session”.

Hope you will find slides useful!

I am very happy to share my slides from the presentation about Apache Hadoop YARN that I gave at the 9th meeting of Warsaw Hadoop User Group 2 days ago.

The slides cover motivation for YARN, description of all YARN daemons, some interesting features and the most interesting configuration parameters and many, many pictures. Hope you will find it useful (please refresh this page if slides do not appear).

I am happy to say the first Hadoop User Group has already been opened in Stockholm, Sweden. We are really excited to see that Hadoop community in Stockholm is so thriving. Actually, the meetup generated pretty big interest and in consequence almost 150 people attended the event!

The meetup was organized in the Cafeteria at Spotify, and two talks were delivered.

I am big fan of football and I really like reading football news. Last week however, I definitely overdid reading it (because Poland played against England in the World Cup 2014 qualifying match). Hopefully, I did realize that it is not the best way to waste my time and today I decided that my next 7 days will be different. Instead, I will read posts from Apache Pig user mailing lists!

The idea is just to read post from the mailing list anytime I feel like reading about football. It means Football zero, Apache Pig hero for me this week ;)