一、点击加载hive表出现如下异常:
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/cli/CliSessionState
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/session/SessionState
解决:将hive lib文件夹下的lib拷贝到kylin lib文件夹下。
查看了下bin里的find-hive-dependency.sh,有设置hive_dependency 包含了所有hive lib下的jar包,但不知道为啥没起作用。所以只好拷贝操作了
在网上看到一个人说,修改bin目录下的kylin.sh
在HBASE_CLASSPATH_PREFIX把hive_dependency 加上。经测可用。
二、解决jar问题,hive表依然加载不出来(点击加载树,一直在转圈)
查看log,如:tail -f -n200 /opt/app/apache-kylin-2.0.0-bin/logs/kylin.log
发现没有异常,弄得我直挠头。尝试点了下reload table按钮,终于报出了异常,但是后台log依然没打出来,报出异常提示连接不上元数据。
hive不知道什么时候挂了,输入: hive –service metastore & 启动一下元数据。终于看到加载的hive表了
为了配置kylin,/etc/profile文件加入如下:
export JAVA_HOME=/opt/app/jdk1.8.0_131
PATH=$PATH:/$JAVA_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
JRE_HOME=$JAVA_HOME/jre
export HADOOP_HOME=/opt/app/hadoop-2.8.0
PATH=$PATH:$HADOOP_HOME/bin:$PATH
export HIVE_HOME=/opt/app/apache-hive-2.1.1-bin
export HCAT_HOME=$HIVE_HOME/hcatalog
export HIVE_CONF=$HIVE_HOME/conf
PATH=$PATH:$HIVE_HOME/bin:$PATH
export HBASE_HOME=/opt/app/hbase-1.3.1
PATH=$PATH:$HBASE_HOME/bin:$PATH
export HIVE_CONF=/opt/app/apache-hive-2.1.1-bin/conf PATH=$PATH:$HIVE_HOME/bin:$PATH export KYLIN_HOME=/opt/app/apache-kylin-2.0.0-bin
PATH=$PATH:$KYLIN_HOME/bin:$PATH export KYLIN_HOME=/opt/app/kylin/
export ZOOKEEPER_HOME=/opt/app/zookeeper-3.4.6
export PATH=$ZOOKEEPER_HOME/bin:$PATH
问题:将hadoop format,重新搭建环境,报异常: org.apache.hadoop.hbase.TableExistsException: kylin_metadata_user
解决:
1.如果是独立的zookeeper
在zookeeper的bin目录下有一个zkCli.sh ,运行进入;
进入后运行 rmr /hbase , 然后quit退出;
2.如果是用hbase自带的zookeeper
用 hbase zkcli进入
进入后运行 rmr /hbase , 然后quit退出;
问题:File does not exist: hdfs://localhost:9000/home/hadoop/data/mapred/staging/root1629090427/.staging/job_local1629090427_0006/libjars/hive-metastore-1.2.1.jar
是由于mapred-site.xml、yarn没有配置正确造成的。
mapred-site.xml
mapreduce.reduce.java.opts -Xms2000m -Xmx4600m mapreduce.map.memory.mb 5120 mapreduce.reduce.input.buffer.percent 0.5 mapreduce.reduce.memory.mb 2048 mapred.tasktracker.reduce.tasks.maximum 2 mapreduce.framework.name yarn mapreduce.jobhistory.address localhost:10020 yarn.app.mapreduce.am.staging-dir /home/hadoop/data/mapred/staging mapreduce.jobhistory.intermediate-done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate mapreduce.jobhistory.done-dir ${yarn.app.mapreduce.am.staging-dir}/history/done
yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler
mapreduce.jobtracker.staging.root.dir /home/hadoop/data/mapred/staging
yarn.app.mapreduce.am.staging-dir /home/hadoop/data/mapred/staging
yarn.resourcemanager.hostname localhost
记得启动 /hadoop-2.8.0/sbin/mr-jobhistory-daemon.sh start historyserver
问题:build cube没有成功,得到如下log
Counters: 41
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=310413
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1700
HDFS: Number of bytes written=0
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Failed map tasks=1
Failed reduce tasks=6
Killed reduce tasks=3
Launched map tasks=2
Launched reduce tasks=9
Other local map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=10602
Total time spent by all reduces in occupied slots (ms)=76818
Total time spent by all map tasks (ms)=10602
Total time spent by all reduce tasks (ms)=76818
Total vcore-seconds taken by all map tasks=10602
Total vcore-seconds taken by all reduce tasks=76818
Total megabyte-seconds taken by all map tasks=10856448
Total megabyte-seconds taken by all reduce tasks=78661632
Map-Reduce Framework
Map input records=0
Map output records=3
Map output bytes=42
Map output materialized bytes=104
Input split bytes=1570
Combine input records=3
Combine output records=3
Spilled Records=3
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=168
CPU time spent (ms)=3790
Physical memory (bytes) snapshot=356413440
Virtual memory (bytes) snapshot=2139308032
Total committed heap usage (bytes)=195035136
File Input Format Counters
Bytes Read=0
在mapred-site.xml中加上如下配置:
mapreduce.reduce.java.opts -Xms2000m -Xmx4600m
mapreduce.map.memory.mb 5120
mapreduce.reduce.input.buffer.percent 0.5
mapreduce.reduce.memory.mb 2048
转自 https://blog.csdn.net/dirful/article/details/74012473