Hadoop for the .NET Developer: Troubleshooting with the MapReduce Job Logs

NOTE This post is one in a series on Hadoop for .NET Developers.

Despite your best efforts, you will occasionally have to deal with failed jobs. To troubleshoot such a job, it helps to understand how to use the logs available to you on the Hadoop cluster. We’ll focus on how to access these logs in this blog post but this is in no way intended to be an exhaustive review of the log information available to you.

To access the logs associated with a job, make note of the job name. This name, in the job_YYYYYMMDDhhmm_NN format, is displayed in the console window as you run your job in Visual Studio. The job name is also available …. If you don’t have access to the job name, then knowing the date and time the job started can help you identify it in the list of jobs recently run on the cluster.

Accessing HTTP port 50030 on the cluster on which the job ran, e.g. https://localhost:50030, will take you to the Job Tracker page. You can access this port directly or find a shortcut to it on the desktop of your HDInsight name node (whether local or in Azure). The Job Tracker page presents a list of running, completed, and retired jobs. Locate your job in the appropriate list and click on the job name to access the job log.

In the job log, you will see a list of tasks followed by a set of counters. In the Task list, you can inspect the various tasks associated with your job by clicking the available links. The task page will present you summary info about your task but also provide you the option to inspect the last 4K, 8K or the entire contents of that task’s log. Digging through the task logs will often help you identify where and why your job is failing. If you wish to submit your own messages to these logs, use the Log property of the context object in your map and reduce functions to do so.

You might also wish to employ the counters presented on the job page to understand what is taking place in your job. By default, MapReduce captures a number of interesting counters which are defined here. You can also create your own counters from within your map and reduce methods by using the context object’s IncrementCounter method. This method allows you to provide the name of a custom counter along with an integer increment. This overloaded method also allows you to define a category for the counter so that if you make use of many custom counters, you can better organize this on the job page.

While this has been a very lightweight exploration of the MapReduce job logs, hopefully it helps you get started with troubleshooting any problems you might encounter.