最近线上的rm crash了一次,查看日志,发现有如下报错:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
2015
-
01
-
05
18
:
00
:
11
,
523
ERROR org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Request
for
appInfo of unknown attemptappattempt_1418702129929_2281200_000001
2015
-
01
-
05
18
:
00
:
11
,
524
ERROR org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Request
for
appInfo of unknown attemptappattempt_1418702129929_2281200_000001
2015
-
01
-
05
18
:
00
:
11
,
526
ERROR org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Request
for
appInfo of unknown attemptappattempt_1418702129929_2281200_000001
2015
-
01
-
05
18
:
00
:
11
,
527
FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type APP_ADDED to the scheduler
org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root already exists!
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:
126
)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:
107
)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:
217
)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics.forQueue(FSQueueMetrics.java:
94
)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.<init>(FSQueue.java:
61
)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.<init>(FSLeafQueue.java:
59
)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.createLeafQueue(QueueManager.java:
199
)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getLeafQueue(QueueManager.java:
144
)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:
581
)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:
998
)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:
108
)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:
378
)
at java.lang.Thread.run(Thread.java:
662
)
|
这其实是yarn的一个bug,在提交的job queue为空时会触发这个bug,导致rm抛出异常而退出,
bug id:https://issues.apache.org/jira/browse/YARN-2863
可以通过如下命令重现:
1
|
yarn jar hadoop-mapreduce-examples-
2.0
.
0
-cdh4.
6.0
.jar pi -Dmapreduce.job.queuename=
20
10
|
在bug 描述中可以看到通过 设置yarn.scheduler.fair.allow-undeclared-pools为false是可以修复这个问题的,但是在cdh4.6.0中是没有这个配置的,可以通过
https://issues.apache.org/jira/browse/YARN-1213
增加一个配置。在queue为空时会使用default的queue。
这个bug在cdh5.2.0已经fix
在5.2.0的yarn中增加了下面两个如下配置
1
2
|
yarn.scheduler.fair.allow-undeclared-pools
//是否允许有不存在的queue,默认为true
yarn.scheduler.fair.user-as-
default
-queue
//当queue不存在时,是否使用用户名来代替default,默认为true
|
本文转自菜菜光 51CTO博客,原文链接:http://blog.51cto.com/caiguangguang/1602231,如需转载请自行联系原作者