Impala安装

一、下载文件（本文选择2.5.0）

地址：http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.7.0/RPMS/x86_64/

二、安装过程

1、安装bigtop-jsvc-0.6.0+cdh5.7.0+804-1.cdh5.7.0.p0.78.el6.x86_64.rpm 

[root@dmp8 impalaRPMs]# rpm -ivh bigtop-jsvc-0.6.0+cdh5.7.0+804-1.cdh5.7.0.p0.78.el6.x86_64.rpm warning: bigtop-jsvc-0.6.0+cdh5.7.0+804-1.cdh5.7.0.p0.78.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY Preparing... [100%] Updating / installing... 1:bigtop-jsvc-0.6.0+cdh5.7.0+804-1. [100%] [root@dmp8 impalaRPMs]# 

2、安装impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64.rpm 

[root@dmp8 impalaRPMs]# rpm -ivh impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64.rpm warning: impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY error: Failed dependencies: hadoop is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 hadoop-hdfs is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 hadoop-yarn is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 hadoop-mapreduce is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 hbase is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 hive >= 0.12.0+cdh5.1.0 is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 zookeeper is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 hadoop-libhdfs is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 avro-libs is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 parquet is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 sentry >= 1.3.0+cdh5.1.0 is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 sentry is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 /lib/lsb/init-functions is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 libhdfs.so.0.0.0()(64bit) is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 libsasl2.so.2()(64bit) is needed by impala-2.5.0+cdh5.7.0+0-1.cdh5.7.0.p0.147.el6.x86_64 [root@dmp8 impalaRPMs]# 

发现缺少很多包；

1）、下载安装avro-libs-1.7.6+cdh5.7.0+113-1.cdh5.7.0.p0.78.el6.noarch.rpm ；

下载地址：http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.7.0/RPMS/noarch/

2）、加权限； chown -R impala:impala  /home/hadoop/hdfs_sockets/dn;(需要的话创建dn目录)

E1120 19:32:05. 1304 logging.cc:120] stderr will be logged to this file. E1120 19:32:08. 1304 impala-server.cc:247] Invalid short-circuit reads configuration: - Impala cannot read or execute the parent directory of dfs.domain.socket.path E1120 19:32:08. 1304 impala-server.cc:249] Aborting Impala Server startup due to improper configuration

查看hdfs-site.xm;

 
   
   
     dfs.domain.socket.path 
    
   
     /home/hadoop/hdfs_sockets/dn 
    
  

impala-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rp缺少依赖包.解压impala_lib.tar.gz,将其中的jar包分别拷入三台服务器的/usr/lib/impala/lib/中并执行

4、把hive-site.xml, core-site.xml, hdfs-site.xml ( 这三个文件都拷贝到 /etc/impala/conf 下 

  hive-site.xml增加：                 
  
             
   
     dfs.datanode.data.dir.perm 
             
   
     755 
         
        
  
             
   
     dfs.block.local-path-access.user 
             
   
     hadoop,impala 
         
   core-site.xml增加：      
  
         
   
     dfs.client.read.shortcircuit 
         
   
     true 
    
   
  
         
   
     dfs.client.read.shortcircuit.skip.checksum 
         
   
     false 
      
  

6、修改三台服务器的/etc/default/impala如下:

IMPALA_CATALOG_SERVICE_HOST=192.168.200.137 IMPALA_STATE_STORE_HOST=192.168.200.137 IMPALA_STATE_STORE_PORT=24000 IMPALA_BACKEND_PORT=22000 IMPALA_LOG_DIR=/var/log/impala IMPALA_SCRATCH_DIR=/home/impalatmp IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} " IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}" IMPALA_SERVER_ARGS=" \     -log_dir=${IMPALA_LOG_DIR} \     -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \     -state_store_port=${IMPALA_STATE_STORE_PORT} \     -use_statestore \     -state_store_host=${IMPALA_STATE_STORE_HOST} \     -be_port=${IMPALA_BACKEND_PORT} \     -scratch_dirs=${IMPALA_SCRATCH_DIR}" ENABLE_CORE_DUMPS=false # LIBHDFS_OPTS=-Djava.library.path=/usr/lib/impala/lib # MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar # IMPALA_BIN=/usr/lib/impala/sbin # IMPALA_HOME=/usr/lib/impala # HIVE_HOME=/usr/lib/hive # HBASE_HOME=/usr/lib/hbase # IMPALA_CONF_DIR=/etc/impala/conf # HADOOP_CONF_DIR=/etc/impala/conf # HIVE_CONF_DIR=/etc/impala/conf # HBASE_CONF_DIR=/etc/impala/conf

注：往/var/lib/impala里拷入mysql-connector-java-5.1.23-bin.jar

红帽系统需要将mysql-connector-java-5.1.23-bin.jar赋予impala用户权限

7.需手工创建/home/impala/tmp否则会报

Create file /tmp/impala-scratch/2e4a5be8bda30756:abd1b7200ec0f09e_1059fb5e-fa2f-4fbe-83e5-6bd577a563a1 failed with errno=2 description=Error(2): No such file or directory

8.启动impala

先在192.168.91.9目录执行: 

service impala-state-store restart

service impala-catalog restart

然后分别在三台服务器上执行：

service impala-server restart

9.查看impala运行状态

执行ps  -ef |grep impala如下所示:

[root@dmp9 impala]# ps -ef|grep impala impala 4721 1 0 20:23 ? 00:00:00 /usr/lib/impala/sbin/statestored -log_dir=/var/log/impala -state_store_port=24000 impala 4827 1 1 20:24 ? 00:00:15 /usr/lib/impala/sbin/catalogd -log_dir=/var/log/impala impala 4987 1 0 20:24 ? 00:00:07 /usr/lib/impala/sbin/impalad -log_dir=/var/log/impala -catalog_service_host=192.168.91.9 -state_store_port=24000 -use_statestore -state_store_host=192.168.91.9 -be_port=22000 root 5817 30925 0 20:45 pts/13 00:00:00 grep --color=auto impala [root@dmp9 impala]# 

说明impala已经成功运行；

10、错误；

1、端口被atlas占用 [root@dmp9 impala]# more impalad.ERROR Log file created at: 2018/11/20 20:17:02 Running on machine: dmp9 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg E1120 20:17:02.022810 3940 logging.cc:120] stderr will be logged to this file. E1120 20:17:04. 4134 thrift-server.cc:176] ThriftServer 'beeswax-frontend' (on port: 21000) exited due to TException: Could not bind: Transport endpoint is not connected E1120 20:17:04. 3940 thrift-server.cc:165] ThriftServer 'beeswax-frontend' (on port: 21000) did not start correctly E1120 20:17:04. 3940 impalad-main.cc:87] ThriftServer 'beeswax-frontend' (on port: 21000) did not start correctly 查看端口被谁占用了： netstat -alnp |grep 21000; [root@dmp4 impala]# netstat -anlp |grep 21000; tcp6 0 0 :::21000 :::* LISTEN 29920/java tcp6 0 0 127.0.0.1:21000 127.0.0.1:42740 ESTABLISHED 29920/java tcp6 0 0 192.168.91.4:42020 192.168.91.4:21000 TIME_WAIT - tcp6 0 0 192.168.91.4:38300 192.168.91.2:21000 TIME_WAIT - tcp6 0 0 192.168.91.4:21000 192.168.91.4:42032 ESTABLISHED 29920/java tcp6 0 0 192.168.91.4:42036 192.168.91.4:21000 ESTABLISHED 30571/java tcp6 0 0 192.168.91.4:42032 192.168.91.4:21000 ESTABLISHED 30031/java tcp6 0 0 127.0.0.1:42740 127.0.0.1:21000 ESTABLISHED 29920/java tcp6 0 0 192.168.91.4:42026 192.168.91.4:21000 TIME_WAIT - tcp6 0 0 192.168.91.4:21000 192.168.91.4:42036 ESTABLISHED 29920/java tcp6 0 0 192.168.91.4:38310 192.168.91.2:21000 ESTABLISHED 2492/java tcp6 0 0 192.168.91.4:42022 192.168.91.4:21000 TIME_WAIT - [root@dmp4 impala]# ps -ef|grep 30571; root 16985 15266 0 11:16 pts/2 00:00:00 grep --color=auto 30571 root 30571 1 0 Oct18 ? 00:48:10 java -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xms1G -Xmx1G -jar /home/zhaozhiwei/system-service-0.0.1/system-service-0.0.1.jar --spring.profiles.active=default 

2、缺失 libpython2.6.so.1.0; 从别的机器下载一个； 放到：/usr/lib64/

3、版本冲突问题 1）rpm -qa|grep hadoop; 查看冲突包； 2）rpm -e --nodeps haoop* 冲突的删除； 3）rpm -ivh --nodeps haoop*.rpm 重新安装； 4）如果rpm -ivh --nodeps haoop*.rpm 报安装了几个包了；则把安装的包移走再批量安装； 

4、libsasl2.so.2缺失； [root@dmp10 impala]# more impala-server.log /usr/lib/impala/sbin/impalad: error while loading shared libraries: libsasl2.so.2: cannot open shared object file: No such file or directory /usr/lib/impala/sbin/impalad: error while loading shared libraries: libsasl2.so.2: cannot open shared object file: No such file or directory 解决方法： libsasl2*:[root@dmp10 impala]# whereis libsasl2.so* libsasl2: /usr/lib64/libsasl2.so [root@dmp10 impala]# cd /usr/lib64/ [root@dmp10 lib64]# ls |grep libsas libsasl2.so libsasl2.so.3 libsasl2.so.3.0.0 建立符号链接： ln -s libsasl2.so.3 libsasl2.so.2;

5、[root@dmp15 impala-cdh5.7-rpm]# service impala-server restart /etc/init.d/impala-server: line 34: /lib/lsb/init-functions: No such file or directory /etc/init.d/impala-server: line 111: killproc: command not found /etc/init.d/impala-server: line 118: log_failure_msg: command not found /etc/init.d/impala-server: line 103: log_success_msg: command not found 解决方法： sudo yum -y install redhat-lsb [root@dmp15 impala-cdh5.7-rpm]# service impala-server restart; Stopped Impala Server: [ OK ] Started Impala Server (impalad): [ OK ] [root@dmp15 impala-cdh5.7-rpm]# 不再报错；

11、针对21000端口被占用，修改Impalad的端口；

vi /etc/default/impala; IMPALA_CATALOG_SERVICE_HOST=192.168.91.9 IMPALA_STATE_STORE_HOST=192.168.91.9 IMPALA_STATE_STORE_PORT=24000 IMPALA_BACKEND_PORT=22000 IMPALA_LOG_DIR=/var/log/impala IMPALA_SCRATCH_DIR=/data1/impala/tmp beeswax_port=21010 #修改21000对应的端口号； IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} " IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}" IMPALA_SERVER_ARGS=" \ -log_dir=${IMPALA_LOG_DIR} \ -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \ -state_store_port=${IMPALA_STATE_STORE_PORT} \ -beeswax_port=${beeswax_port} \ #指定下端口； -use_statestore \ -state_store_host=${IMPALA_STATE_STORE_HOST} \ -be_port=${IMPALA_BACKEND_PORT}" ENABLE_CORE_DUMPS=false # LIBHDFS_OPTS=-Djava.library.path=/usr/lib/impala/lib # MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar # IMPALA_BIN=/usr/lib/impala/sbin # IMPALA_HOME=/usr/lib/impala # HIVE_HOME=/usr/lib/hive # HBASE_HOME=/usr/lib/hbase # IMPALA_CONF_DIR=/etc/impala/conf # HADOOP_CONF_DIR=/etc/impala/conf # HIVE_CONF_DIR=/etc/impala/conf # HBASE_CONF_DIR=/etc/impala/conf 

登录时需要指定端口：

impala-shell -i dmp9:21010;

三、部署位置；

1、mpala-state-store,impala-catalog  放在任意一个namenode; 

2、impala-server 放在每个datanode;