hadoop伪分布式安装部署HIVE

  1. 云栖社区>
  2. 博客>
  3. 正文

hadoop伪分布式安装部署HIVE

wsc449 2017-11-13 16:59:37 浏览534
展开阅读全文

应用场景

当我们按照hadoop伪分布式集群搭建博客搭建了hadoop以后,发现这是一个空的hadoop,只有YARN,MapReduce,HDFS,而这些实际上我们一般不会直接使用,而是需要另外部署Hadoop的其他组件,来辅助使用。比如我们把数据存储到了hdfs,都是文件格式,用起来肯定不方便,用HIVE把数据从HDFS映射成表结构,直接用sql语句即可操作数据。另外针对分布式数据计算算法MapReduce,需要直接写MapReduce程序,比较复杂,此时使用Hive,就可以通过写SQL语句,来实现MapReduce的功能实现。

操作步骤

1. 安装mysql

安装mysql,配置远程登录,具体参考mysql章节
并配置可以远程登录mysql

2. 下载hive2.1并解压配置环境变量

下载hive2.1

解压到opt目录下,更换目录为hive-2.1.1
 # tar zxvf apache-hive-2.1.1-bin.tar.gz
 # mv apache-hive-2.1.1-bin hive-2.1.1

配制系统环境变量:

 # vim /etc/profile

export HADOOP_HOME=/usr/local/hadoop-2.6.0
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"
export HIVE_HOME=/opt/hive-2.1.1
export HIVE_CONF_DIR=${HIVE_HOME}/conf
export JAVA_HOME=/opt/jdk1.7.0_79
export PATH=.:$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$PATH

保存后,使配制生效:
# source /etc/profile

3. 修改hive-env.sh文件

修改hive-env.sh文件,在文件底部增加以下环境变量:

 # cd /opt/hive-2.1.1/conf
 # cp hive-env.sh.template hive-env.sh

 # vim hive-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_79
export HIVE_HOME=/opt/hive-2.1.1
export HIVE_CONF_DIR=/opt/hive-2.1.1/conf
export HIVE_AUX_JARS_PATH=/opt/hive-2.1.1/lib
export HADOOP_HOME=/opt/hadoop-2.6.0

4. 修改hive-site.xml文件

4.1 修改hive-site.xml文件,并将所有的${system:java.io.tmpdir}替换为绝对路径:

 # cp hive-default.xml.template hive-site.xml
 # vim hive-site.xml 

由于在该配置文件中有如下两个配置项注明了hive在HDFS中数据存储的目录,因此我们需要在HDFS上手动创建并赋权限,也就是需要在hdfs上创建/tmp/hive 和/user/hive/warehouse

 # hadoop fs -mkdir -p /user/hive/warehouse
 # hadoop fs -chmod -R 777 /user/hive/warehouse #递归赋予读写权限 
 # hadoop fs -mkdir -p /tmp/hive/ #创建/tmp/hive/目录  
 # hadoop fs -chmod -R 777 /tmp/hive #目录赋予读写权限   
 <property>  
  <name>hive.exec.local.scratchdir</name>  
  <value>${system:java.io.tmpdir}/${system:user.name}</value>  
  <description>Local scratch space for Hive jobs</description>  
</property> 

<property>  
  <name>hive.downloaded.resources.dir</name>  
 <value>${system:java.io.tmpdir}/${hive.session.id}_resources</value>
  <description>Temporary local directory for added resources in the remote file system.</description>  
</property>  

<property>  
  <name>hive.querylog.location</name>  
  <value>${system:java.io.tmpdir}/${system:user.name}</value>  
  <description>Location of Hive run time structured log file</description>  
</property>  

 <property>  
  <name>hive.server2.logging.operation.log.location</name>  
  <value>${system:java.io.tmpdir}/${system:user.name}/operation_logs</value>  
  <description>Top level directory where operation logs are stored if logging functionality is enabled</description>  
</property>  

替换后:

 <property>  
  <name>hive.exec.local.scratchdir</name>  
  <value>/opt/hive-2.1.1/tmp/</value>  
  <description>Local scratch space for Hive jobs</description>  
</property>  

<property>  
  <name>hive.downloaded.resources.dir</name>  
  <value>/opt/hive-2.1.1/tmp/${hive.session.id}_resources</value>  
  <description>Temporary local directory for added resources in the remote file system.</description>  
</property>  

<property>  
  <name>hive.querylog.location</name>  
  <value>/opt/hive-2.1.1/tmp/</value>  
  <description>Location of Hive run time structured log file</description>  
</property>  

<property>  
  <name>hive.server2.logging.operation.log.location</name>  
  <value>/opt/hive-2.1.1/tmp/root/operation_logs</value>  
  <description>Top level directory where operation logs are stored if logging functionality is enabled</description>  
</property>  

4.2 修改hive-site.xml文件,修改文件中的元数据的连接,驱动,用户名,密码

hive-site.xml中相关元数据信息配制:
javax.jdo.option.ConnectionDriverName,将对应的value修改为MySQL驱动类路径;
javax.jdo.option.ConnectionURL,将对应的value修改为MySQL的地址;
javax.jdo.option.ConnectionUserName,将对应的value修改为MySQL数据库登录名;
javax.jdo.option.ConnectionPassword,将对应的value修改为MySQL数据库的登录密码:

  <property>  
    <name>javax.jdo.option.ConnectionDriverName</name>  
    <value>com.mysql.jdbc.Driver</value>  
    <description>Driver class name for a JDBC metastore</description>
  </property> 

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://192.168.208.110:3306/hive?createDatabaseIfNotExist=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    <description>Username to use against metastore database</description>
  </property> 

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>11111</value>
    <description>password to use against metastore database</description>
  </property>

将MySQL驱动包添加到Hive的lib目录下:mysql-connector-java-5.1.38-bin.jar

5. hive的启动及测试:

对hive元数据初始化(mysql中hive元信息初始化、建表等):

 # schematool -initSchema -dbType mysql
 # hive

6. 注意

注:如果要用beeline -u jdbc:hive2://ip:10000进行连接,需要修改hdfs的core-site.xml配置

<property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
</property>

# 备注:hadoop.proxyuser.XXX.hosts 与 hadoop.proxyuser.XXX.groups 中XXX为异常信息中User:* 中的用户名部分,这里用户名都是root,所以hdfs的core-site.xml中的标签名为hadoop.proxyuser.root.hosts与hadoop.proxyuser.root.groups

网友评论

登录后评论
0/500
评论
wsc449
+ 关注