李林超博客
首页
归档
留言
友链
动态
关于
归档
留言
友链
动态
关于
首页
大数据
正文
07.Yarn生产环境核心参数配置案例
Leefs
2021-09-10 AM
1121℃
0条
# 07.Yarn生产环境核心参数配置案例 ### 前言 本篇通过一个案例来分析生产环境中,Yarn需要修改的配置参数。 ### 一、案例 #### 1.1 需求 > 从1G数据中,统计每个单词出现次数。服务器3台,每台配置4G内存,4核CPU,4线程。 #### 1.2 需求分析 ``` 1G / 128m = 8个MapTask;1个ReduceTask;1个mrAppMaster 平均每个节点运行10个 / 3台 ≈ 3个任务(4 3 3) ``` #### 1.3 修改yarn-site.xml配置参数 ```xml
The class to use as the resource scheduler.
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
Number of threads to handle scheduler interface.
yarn.resourcemanager.scheduler.client.thread-count
8
Enable auto-detection of node capabilities such as memory and CPU.
yarn.nodemanager.resource.detect-hardware-capabilities
false
Flag to determine if logical processors(such as hyperthreads) should be counted as cores. Only applicable on Linux when yarn.nodemanager.resource.cpu-vcores is set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true.
yarn.nodemanager.resource.count-logical-processors-as-cores
false
Multiplier to determine how to convert phyiscal cores to vcores. This value is used if yarn.nodemanager.resource.cpu-vcores is set to -1(which implies auto-calculate vcores) and yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The number of vcores will be calculated as number of CPUs * multiplier.
yarn.nodemanager.resource.pcores-vcores-multiplier
1.0
Amount of physical memory, in MB, that can be allocated for containers. If set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically calculated(in case of Windows and Linux). In other cases, the default is 8192MB.
yarn.nodemanager.resource.memory-mb
4096
Number of vcores that can be allocated for containers. This is used by the RM scheduler when allocating resources for containers. This is not used to limit the number of CPUs used by YARN containers. If it is set to -1 and yarn.nodemanager.resource.detect-hardware-capabilities is true, it is automatically determined from the hardware in case of Windows and Linux. In other cases, number of vcores is 8 by default.
yarn.nodemanager.resource.cpu-vcores
4
The minimum allocation for every container request at theRM in MBs. Memory requests lower than this will be set to the value of this property. Additionally, a node manager that is configured to have less memory than this value will be shut down by the resource manager.
yarn.scheduler.minimum-allocation-mb
1024
The maximum allocation for every container request at the RM in MBs. Memory requests higher than this will throw an InvalidResourceRequestException.
yarn.scheduler.maximum-allocation-mb
2048
The minimum allocation for every container request at the RM in terms of virtual CPU cores. Requests lower than this will be set to the value of this property. Additionally, a node manager that is configured to have fewer virtual cores than this value will be shut down by the resource manager.
yarn.scheduler.minimum-allocation-vcores
1
The maximum allocation for every container request at the RM in terms of virtual CPU cores. Requests higher than this will throw an InvalidResourceRequestException.
yarn.scheduler.maximum-allocation-vcores
2
Whether virtual memory limits will be enforced for containers.
yarn.nodemanager.vmem-check-enabled
false
Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio.
yarn.nodemanager.vmem-pmem-ratio
2.1
``` **关闭虚拟内存检查原因** ![07.Yarn生产环境核心参数配置案例01.jpg](https://lilinchao.com/usr/uploads/2021/09/125297225.jpg) ```xml
Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio.
yarn.nodemanager.vmem-pmem-ratio
2.1
``` #### 1.4 分发配置 *注意:如果集群的硬件资源不一致,要每个NodeManager单独配置* #### 1.5 重启集群 ``` [root@hadoop102 hadoop-3.1.3]# sbin/stop-yarn.sh [root@hadoop103 hadoop-3.1.3]# sbin/start-yarn.sh ``` #### 1.6 执行WordCount程序 ``` [root@hadoop102 hadoop-3.1.3]# hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output ``` #### 1.7 观察Yarn任务执行页面 `http://hadoop103:8088/cluster/apps` *附:* *文章来源《尚硅谷大数据之Hadoop》*
标签:
Hadoop
,
Yarn
非特殊说明,本博所有文章均为博主原创。
如若转载,请注明出处:
https://www.lilinchao.com/archives/1431.html
上一篇
06.Yarn核心参数配置
下一篇
08.Yarn容量调度器多队列配置案例
取消回复
评论啦~
提交评论
栏目分类
随笔
2
Java
326
大数据
229
工具
31
其它
25
GO
47
标签云
Hive
FileBeat
Elasticsearch
Spark SQL
Java工具类
SpringCloudAlibaba
Elastisearch
字符串
ClickHouse
数学
RSA加解密
NIO
GET和POST
MySQL
栈
Typora
Hbase
Java编程思想
Hadoop
Flume
Java阻塞队列
排序
国产数据库改造
Jenkins
递归
Netty
Linux
Spark Core
Shiro
Java
友情链接
申请
范明明
庄严博客
Mx
陶小桃Blog
虫洞