TEZ 0.5以后得版本和hive0.13不兼容
[hadoop@master tez]$ hadoop fs -mkdir /tez16/04/04 06:47:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
[hadoop@master tez]$ hadoop fs -put ../software/tez-0.7.0.tar.gz /tez16/04/04 06:48:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
[hadoop@master tez]$ hadoop fs -ls /tez3、需要在hadoop的master节点上面的 ${HADOOP_HOME}/etc/hadoop目录下面创建一个tez-site.xml文件，里面填写如下内容
<configuration> <property> <name>tez.lib.uris
name> <value>${fs.defaultFS}/tez/tez-0.7.0.tar.gz
value>
property>
configuration>以上完成了tez的基本配置，接下来有2中方法使我们的任务运行在tez上，
export TEZ_HOME=/oneapm/local/tez-0.5.3 for jar in `ls $TEZ_HOME |grep jar`; do export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_HOME/$jar done for jar in `ls $TEZ_HOME/lib`; do export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_HOME/lib/$jar done在HADOOP_CLASSPATH变量声明复制后加入如上代码,将tez的所有jar包添加到hadoop_classpath中 1、2 编辑mapred-site.xml <property> <name>mapreduce.framework.name
name> <value>yarn-tez
value>
property>在hadoop上提交任务
[hadoop@master tez]$ hadoop jar tez-examples-0.7.0.jar orderedwordcount /input1/EMP /output3 2、1将tez安装目录和tez安装目录下来的lib目录的jar全部拷贝到 ${HIVE_HOME}/lib目录下面
[hadoop@master tez]$ cp tez-*.jar ../apache-hive-0.13.0-bin/lib/ [hadoop@master lib]$ cp *.jar ~/apache-hive-0.13.0-bin/lib/[hadoop@master tez]$ cp *.jar ~/hive/apache-hive-0.14.0-bin/lib/ [hadoop@master tez]$ cp ./lib/*.jar ~/hive/apache-hive-0.14.0-bin/lib/如果执行hive语句显示的是类似下面的说明TEZ on hive以后成功了
hive> set hive.execution.engine=tez; hive> select count(1) from xiaomi; Query ID = hadoop_252_0fa4e612-3b2f-468a-8d61-d700c96105d3 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_67_0002) -------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 .......... SUCCEEDED 1 1 0 0 0 0 Reducer 2 ...... SUCCEEDED 1 1 0 0 0 0 -------------------------------------------------------------------------------- VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 10.96 s -------------------------------------------------------------------------------- OK 300 Time taken: 23.82 seconds, Fetched: 1 row(s)如果你想运行任务在mapreduce上面，你可以设置 set hive.execution.engine=mr;
根据如上的配置,集群已经是tez模式了,通过set hive.execution.engine=mr; 似乎没有什么作用,8088任务列表的application type依然为tez
发布者：全栈程序员-站长，转载请注明出处：https://javaforall.net/228293.html原文链接：https://javaforall.net
