首页 \ 问答 \ 在YARN上运行时，Spark调度程序池如何工作？(How do Spark scheduler pools work when running on YARN?)

在YARN上运行时，Spark调度程序池如何工作？(How do Spark scheduler pools work when running on YARN?)

 我在YARN（Hadoop 2.6.0 / CDH 5.5）上部署了各种Spark版本（1.6,2.0,2.1）。 我试图保证某个应用程序永远不会在我们的YARN集群上缺乏资源，无论在那里运行的是什么。  
 我启用了shuffle服务并设置了一些Fair Scheduler Pools ，如Spark文档中所述。 我为高优先级应用程序创建了一个单独的池，我希望永远不会缺乏资源，并给它一个minShare的资源共享：  
<?xml version="1.0"?>
<allocations>
  <pool name="default">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>0</minShare>
  </pool>
  <pool name="high_priority">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>24</minShare>
  </pool>
</allocations>
 
 当我在YARN集群上运行Spark应用程序时，我可以看到我配置的池被识别：  
17/04/04 11:38:20 INFO scheduler.FairSchedulableBuilder: Created pool default, schedulingMode: FAIR, minShare: 0, weight: 1
17/04/04 11:38:20 INFO scheduler.FairSchedulableBuilder: Created pool high_priority, schedulingMode: FAIR, minShare: 24, weight: 1
 
 但是，我没有看到我的应用程序正在使用新的high_priority池，即使我在调用spark-submit时设置了spark.scheduler.pool 。 这意味着当群集与常规活动挂钩时，我的高优先级应用程序无法获得所需的资源：  
17/04/04 11:39:49 INFO cluster.YarnScheduler: Adding task set 0.0 with 1 tasks
17/04/04 11:39:50 INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_0 tasks to pool default
17/04/04 11:39:50 INFO spark.ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 1)
17/04/04 11:40:05 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
 
 我在这里想念的是什么？ 我的同事和我试图在YARN中实现先发制人，但这没有做任何事情。 然后我们意识到YARN中的概念与称为YARN队列的 Spark调度程序池非常相似。 所以现在我们不确定这两个概念是否会以某种方式发生冲突。  
 我们如何让我们的高优先级池按预期工作？ Spark调度程序池和YARN队列之间是否存在某种冲突？ 

I have a mix of Spark versions (1.6, 2.0, 2.1) all deployed on YARN (Hadoop 2.6.0 / CDH 5.5). I'm trying to guarantee that a certain application will never be starved of resources on our YARN cluster, regardless of what else may be running on there. 
I've enabled the shuffle service and setup some Fair Scheduler Pools as described in the Spark docs. I created a separate pool for the high priority application I want never to be starved of resources, and gave it a minShare of resources: 
<?xml version="1.0"?>
<allocations>
  <pool name="default">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>0</minShare>
  </pool>
  <pool name="high_priority">
    <schedulingMode>FAIR</schedulingMode>
    <weight>1</weight>
    <minShare>24</minShare>
  </pool>
</allocations>
 
When I run a Spark application on our YARN cluster, I can see that the pools I configured are recognized: 
17/04/04 11:38:20 INFO scheduler.FairSchedulableBuilder: Created pool default, schedulingMode: FAIR, minShare: 0, weight: 1
17/04/04 11:38:20 INFO scheduler.FairSchedulableBuilder: Created pool high_priority, schedulingMode: FAIR, minShare: 24, weight: 1
 
However, I don't see that my application is using the new high_priority pool, even though I am setting spark.scheduler.pool in my call to spark-submit. So that means when the cluster is pegged by regular activity, my high priority application is not getting the resources it needs: 
17/04/04 11:39:49 INFO cluster.YarnScheduler: Adding task set 0.0 with 1 tasks
17/04/04 11:39:50 INFO scheduler.FairSchedulableBuilder: Added task set TaskSet_0 tasks to pool default
17/04/04 11:39:50 INFO spark.ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 1)
17/04/04 11:40:05 WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
 
What am I missing here? My coworkers and I tried enabling preemption in YARN, but that didn't do anything. And then we realized that there is a concept in YARN very similar to Spark scheduler pools called YARN queues. So now we're not sure if the two concepts conflict somehow. 
How can we get our high priority pool to work as expected? Is there some kind of conflict between Spark scheduler pools and YARN queues?

原文：https://stackoverflow.com/questions/43239921

更新时间：2023-04-03 17:04

最满意答案

 这应该是（假设你的字符串存储在' $string '中）：  
my ($var1, $var2) = $string =~ /_(\d+)_(\d+)/s; 
 
 这个想法是抓住数字，直到你得到一个非数字字符：这里' _ '。  
 然后将每个捕获组分配给它们各自的变量。  
 
 正如在这个问题中提到的（以及Kaoru 在下面的评论中）：  
 
  \d确实可以匹配超过10个不同的字符，如果应用于Unicode字符串。  
 
 所以你可以使用：  
my ($var1, $var2) = $string =~ /_([0-9]+)_([0-9]+)/s; 

That should be (assuming your string is stored in '$string'): 
my ($var1, $var2) = $string =~ /_(\d+)_(\d+)/s; 
 
The idea is to grab numbers until you get a non-number character: here '_'. 
Each capturing group is then assign to their respective variable.  
 
As mentioned in this question (and in the comments below by Kaoru): 
 
 \d can indeed match more than 10 different characters, if applied to Unicode strings. 
 
So you can use instead: 
my ($var1, $var2) = $string =~ /_([0-9]+)_([0-9]+)/s;

在YARN上运行时，Spark调度程序池如何工作？(How do Spark scheduler pools work when running on YARN?)

最满意答案

相关问答

Perl将正则表达式匹配组分配给变量(Perl assign regex match groups to variables)[2022-10-11]

将多个正则表达式匹配中的一个分配给变量为Perl one-liner（dereference array？）(Assign one of multiple regex matches to variable as Perl one-liner (dereference array?))[2023-02-06]

如何使用bash将OUTPUT值中的此正则表达式匹配分配给变量？(How do I assign this regex match from an OUTPUT value to a variable using bash?)[2021-11-09]

将jquery正则表达式匹配中的匹配值分配给字符串变量(assign matched values from jquery regex match to string variable)[2024-03-09]

Perl，将正则表达式匹配分配给标量(Perl, Assign regex match to scalar)[2023-12-27]

正则表达式将多个组分配给关键字(Regex assigning multiple groups to keyword)[2023-12-05]

perl正则表达式匹配相同的行(perl regex matching same line)[2022-06-02]

如何将正则表达式分配给变量以供以后在字符串中使用？(How to assign regex to variable for later use on a string?)[2022-04-21]

Javascript：测试正则表达式并分配给变量，如果它在一行中匹配(Javascript: test regex and assign to variable if it matches in one line)[2022-02-23]

对数据使用正则表达式并将匹配项分配给新范围(Use regex on data and assign matches to new scope)[2022-02-12]

相关文章

最新问答