采用ToolRunner执行Hadoop基本面分析程序

A utility to help run Tools.

ToolRunner can be used to run classes implementing Tool interface. It works in conjunction with GenericOptionsParser to parse the generic hadoop command line arguments and modifies the Configuration of the Tool. The application-specific options are passed along without being modified.

run

public static int run(Configuration conf,
                      Tool tool,
                      String[] args)               throws Exception

Runs the given Tool by 
Tool.run(String[]), after parsing with the given generic arguments. Uses the given Configuration, or builds one if null. Sets the Tool’s configuration with the possibly modified version of the conf.

Parameters:

conf – Configuration for the Tool.

tool – Tool to run.

args – command-line arguments to the tool.

Returns:

exit code of the 
Tool.run(String[]) method.

Throws:

public static int run(Tool tool, String[] args) throws Exception

Runs the Tool with its Configuration. Equivalent to run(tool.getConf(), tool, args).

Parameters:

tool – Tool to run.

args – command-line arguments to the tool.

Returns:

exit code of the 
Tool.run(String[]) method.

Throws:

它们均是静态方法。即能够通过类名调用。

除此以外，另一个方法：

static void printGenericCommandUsage(PrintStream out) 
          Prints generic command-line argurments and usage information.

4、ToolRunner完毕下面2个功能：

（1）为Tool创建一个Configuration对象。

（2）使得程序能够方便的读取參数配置。

ToolRunner完整源码例如以下：

package org.apache.hadoop.util;

import java.io.PrintStream;

import org.apache.hadoop.conf.Configuration;

/**
 * A utility to help run {@link Tool}s.
 * 
 * <p><code>ToolRunner</code> can be used to run classes implementing 
 * <code>Tool</code> interface. It works in conjunction with 
 * {@link GenericOptionsParser} to parse the 
 * <a href="{@docRoot}/org/apache/hadoop/util/GenericOptionsParser.html#GenericOptions">
 * generic hadoop command line arguments</a> and modifies the 
 * <code>Configuration</code> of the <code>Tool</code>. The 
 * application-specific options are passed along without being modified.
 * </p>
 * 
 * @see Tool
 * @see GenericOptionsParser
 */
public class ToolRunner {
 
  /**
   * Runs the given <code>Tool</code> by {@link Tool#run(String[])}, after 
   * parsing with the given generic arguments. Uses the given 
   * <code>Configuration</code>, or builds one if null.
   * 
   * Sets the <code>Tool</code>'s configuration with the possibly modified 
   * version of the <code>conf</code>.  
   * 
   * @param conf <code>Configuration</code> for the <code>Tool</code>.
   * @param tool <code>Tool</code> to run.
   * @param args command-line arguments to the tool.
   * @return exit code of the {@link Tool#run(String[])} method.
   */
  public static int run(Configuration conf, Tool tool, String[] args) 
    throws Exception{
    if(conf == null) {
      conf = new Configuration();
    }
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    //set the configuration back, so that Tool can configure itself
    tool.setConf(conf);
    
    //get the args w/o generic hadoop args
    String[] toolArgs = parser.getRemainingArgs();
    return tool.run(toolArgs);
  }
  
  /**
   * Runs the <code>Tool</code> with its <code>Configuration</code>.
   * 
   * Equivalent to <code>run(tool.getConf(), tool, args)</code>.
   * 
   * @param tool <code>Tool</code> to run.
   * @param args command-line arguments to the tool.
   * @return exit code of the {@link Tool#run(String[])} method.
   */
  public static int run(Tool tool, String[] args) 
    throws Exception{
    return run(tool.getConf(), tool, args);
  }
  
  /**
   * Prints generic command-line argurments and usage information.
   * 
   *  @param out stream to write usage information to.
   */
  public static void printGenericCommandUsage(PrintStream out) {
    GenericOptionsParser.printGenericCommandUsage(out);
  }
  
}

Unless explicitly turned off, Hadoop by default specifies two resources, loaded in-order from the classpath:

1. core-default.xml : Read-only defaults for hadoop.
2. core-site.xml: Site-specific configuration for a given hadoop installation.

static{
    //print deprecation warning if hadoop-site.xml is found in classpath
    ClassLoader cL = Thread.currentThread().getContextClassLoader();
    if (cL == null) {
      cL = Configuration.class.getClassLoader();
    }
    if(cL.getResource("hadoop-site.xml")!=null) {
      LOG.warn("DEPRECATED: hadoop-site.xml found in the classpath. " +
          "Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, "
          + "mapred-site.xml and hdfs-site.xml to override properties of " +
          "core-default.xml, mapred-default.xml and hdfs-default.xml " +
          "respectively");
    }
    addDefaultResource("core-default.xml");
    addDefaultResource("core-site.xml");
  }

Configuration.java的源码中包括了以上代码，即通过静态语句为程序载入core-default.xml以及core-site.xml中的參数。

同一时候，检查是否还存在hadoop-site.xml，若还存在，则给出warning，提醒此配置文件已经废弃。

for (Entry<String, String> entry : conf)｛.....｝

（四）关于Tool

package org.apache.hadoop.util;import org.apache.hadoop.conf.Configurable;public interface Tool extends Configurable {    int run(String [] args) throws Exception;}

由此可见，Tool自身仅仅有一个方法run(String[])，同一时候它继承了Configuable的2个方法。

package org.apache.hadoop.conf;

public interface Configurable {

  void setConf(Configuration conf);

  Configuration getConf();
}

2、Configured的源文件例如以下：

package org.apache.hadoop.conf;public class Configured implements Configurable {  private Configuration conf;  public Configured() {    this(null);  }   public Configured(Configuration conf) {    setConf(conf);  }  public void setConf(Configuration conf) {    this.conf = conf;  }  public Configuration getConf() {    return conf;  }}

它有2个构造方法。各自是带Configuration參数的方法与不还參数的方法。