好贷网好贷款

ubuntu配置hive

发布时间:2016-12-3 8:19:12 编辑:www.fx114.net 分享查询网我要评论
本篇文章主要介绍了"ubuntu配置hive",主要涉及到ubuntu配置hive方面的内容,对于ubuntu配置hive感兴趣的同学可以参考一下。

http://www.cnblogs.com/ggjucheng/archive/2012/04/26/2471341.html 安装环境:           机器 只需要安装一台机器       操作系统:Ubuntu 11.04 64操作系统       hadoop:版本是1.0.2,安装在/usr/local/hadoop       sun jdk:版本是1.6.0_31 64bit,安装在/usr/local/jdk       hive:版本是0.8.1,安装在/usr/local/hive 安装步骤: 1.下载       下载hive:http://labs.mop.com/apache-mirror/hive/hive-0.8.1/hive-0.8.1.tar.gz 2.安装 (1)上传hive安装包到机器上,使用root用户登陆:    tar -xvf hive-0.8.1.tar.gz (2)将解压的hive分别移动并改名为/usr/local/hive rm -rf /usr/local/hive mv hive-0.8.1 /usr/local/hive   3.配置hive (1)修改/usr/local/hive/bin/hive-config.sh     在文件末尾加入 export JAVA_HOME=/usr/local/jdk export HIVE_HOME=/usr/local/hive export HADOOP_HOME=/usr/local/hadoop (2) 根据hive-default.xml复制hive-site.xml cp /usr/local/hive/conf/hive-default.xml /usr/local/hive/conf/hive-site.xml (3)配置hive-site.xml,主要配置项如下:         hive.metastore.warehouse.dir:(HDFS上的)数据目录         hive.exec.scratchdir:(HDFS上的)临时文件目录         hive.metastore.warehouse.dir默认值是/user/hive/warehouse         hive.exec.scratchdir默认值是/tmp/hive-${user.name}         以上是默认值,暂时不改。 (4)改变 /usr/local/hive的目录所有者为hadoop chown -R hadoop:hadoop /usr/local/hive (5)配置hive的log4j:       cp /usr/loca/hive/conf/hive-log4j.properties.template  /usr/loca/hive/conf/hive-log4j.properties    修改/usr/loca/hive/conf/hive-log4j.properties将org.apache.hadoop.metrics.jvm.EventCounter改为org.apache.hadoop.log.metrics.EventCounter (6)启动hive       使用hadoop用户登陆,执行/usr/local/hive/bin/hive (7)测试hive hive> create TABLE pokes( id INT, name string); hive> SHOW TABLES; hive> select * from pokes; hive> drop table pokes; 4.优化hive 默认meta数据库为derby ,为了避免使用默认的Derby数据库(有并发访问和性能的问题),通常还需要配置元数据库为MySQL 修改配置文件conf/hive-site.xml  <property> <name>hive.metastore.local</name> <value>false</value> <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://mysql_server_host:3306/hivedb?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=latin1</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>mysql_username</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>mysql_password</value> <description>password to use against metastore database</description> </property> <property> <name>hive.stats.dbconnectionstring</name> <value>jdbc:mysql://mysql_server_host:3306/hive_stats?useUnicode=true&amp;characterEncoding=latin1&amp;user=mysql_username&amp;password=mysql_password&amp;createDatabaseIfNotExist=true</value> <description>The default connection string for the database that stores temporary hive statistics.</description> </property> <property> <name>hive.stats.dbconnectionstring</name> <value>jdbc:mysql://mysql_server_host:3306/hive_stats?useUnicode=true&amp;characterEncoding=utf8&amp;user=mysql_username&amp;password=mysql_password&amp;createDatabaseIfNotExist=true</value> <description>The default connection string for the database that stores temporary hive statistics.</description> </property> <property> <name>hive.stats.dbclass</name> <value>jdbc:mysql</value> <description>The default database that stores temporary hive statistics.</description> </property> <property> <name>hive.stats.jdbcdriver</name> <value>com.mysql.jdbc.Driver</value> <description>The JDBC driver for the database that stores temporary hive statistics.</description> </property> <property> <name>hive.metastore.uris</name> <value>thrift://127.0.0.1:9083</value> </property> 添加metastore启动脚本bin/hive-metastore.sh #!/bin/sh nohup ./hive --service metastore >> metastore.log 2>&1 & echo $! > hive-metastore.pid 添加hive server启动脚本bin/hive-server.sh nohup ./hive --service hiveserver >> hiveserver.log 2>&1 & echo $! > hive-server.pid 启动metastore和hive server ./hive-metastore.sh ./hive-server.sh 使用客户端连接和测试 参考之前的hive测试步骤一一执行。 如果出现 FAILED: Error in metadata: javax.jdo.JDODataStoreException: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception. NestedThrowables: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask 或者是 FAILED: Error in metadata: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask 以上错误,主要是在select操作、drop操作时,就会出错。 这样的错误,可以修改hivedb数据库编码: mysql> alter database hivedb character set latin1; 然后重启hive metastore和hive server就可以了 mysql数据库的编码目前最好设置为latin1,否则使用hive会出现莫名其妙的问题

上一篇:【浅先生wifi热点管理工具】
下一篇:iis修复工具下载

相关文章

关键词: ubuntu配置hive

相关评论