首页 \ 教程 \ hadoop

知识点

hadoop

Hadoop集群默认配置和常用配置

Hadoop 原生态支持常用 InputFormat OutFormat 详解

Hadoop 创建用户及HDFS权限，HDFS操作等常用Shell命令

Eclipse Hadoop环境配置

Hadoop Eclipse 配置

Eclipse下配置使用Hadoop插件

Hadoop 0.20.2 集群配置

Hadoop 配置笔记

Ubuntu下Hadoop环境的配置

Hadoop 参数配置优化

Hadoop 集群配置

云计算Hadoop配置

Hadoop安装以及配置

Eclipse下配置Hadoop环境

Hadoop的thrift server配置

Hadoop常用配置总结

2019-03-28 13:25|来源: 网络

NameNode管理界面：http://namenode:50070

JobTracker管理界面：http://jobtracker:50030

Hadoop守护进程日志存放目录：可以用环境变量${Hadoop_LOG_DIR}进行配置，默认情况下是${HADOOP_HOME}/logs

1．配置类型节点的环境变量

在配置集群的时候可以在conf/hadoop-env.sh配置不同节点的环境变量：

Daemon	Configure Options
NameNode	HADOOP_NAMENODE_OPTS
DataNode	HADOOP_DATANODE_OPTS
SecondaryNamenode	HADOOP_SECONDARYNAMENODE_OPTS
JobTracker	HADOOP_JOBTRACKER_OPTS
TaskTracker	HADOOP_TASKTRACKER_OPTS

例如，可以在hadoop-env.sh中加入下面一行，使NameNode使用ParallelGC

export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}"

2. 配置Hadoop守护进程
conf/core-site.xml:通过该配置文件配置文件系统根目录，即hdfs://namenode

Parameter	Value	Notes
fs.default.name	URI of NameNode.	hdfs://hostname/

conf/hdfs-site.xml:通过该配置文件，配置NameNode和DataNode数据文件的存放目录

Parameter	Value	Notes
dfs.name.dir	NameNode上存放命名空间和日志的本地目录，即fsimage和edits文件的存放目录	如果该配置有由逗号分开的多条目录，那么NameNode会在每条目录中进行冗余存储
dfs.data.dir	DataNode中存放block的目录	如果是由逗号分开的多条目录，则所有的目录都用来存储数据

conf/mapred-site.xml:可以通过这个文件配置MapReduce框架

Parameter	Value	Notes
mapred.job.tracker	JobTracker的IP和端口	host:portpair.
mapred.system.dir	Path on the HDFS where where the MapReduce framework stores system files e.g. /hadoop/mapred/system/.	This is in the default filesystem (HDFS) and must be accessible from both the server and client machines.
mapred.local.dir	Comma-separated list of paths on the local filesystem where temporary MapReduce data is written.	Multiple paths help spread disk i/o.
mapred.tasktracker.{map\|reduce}.tasks.maximum	The maximum number of MapReduce tasks, which are run simultaneously on a given TaskTracker, individually.	Defaults to 2 (2 maps and 2 reduces), but vary it depending on your hardware.
dfs.hosts/dfs.hosts.exclude	List of permitted/excluded DataNodes.	If necessary, use these files to control the list of allowable datanodes.
mapred.hosts/mapred.hosts.exclude	List of permitted/excluded TaskTrackers.	If necessary, use these files to control the list of allowable TaskTrackers.
mapred.queue.names	Comma separated list of queues to which jobs can be submitted.	The MapReduce system always supports atleast one queue with the name as default. Hence, this parameter's value should always contain the string default. Some job schedulers supported in Hadoop, like the Capacity Scheduler, support multiple queues. If such a scheduler is being used, the list of configured queue names must be specified here. Once queues are defined, users can submit jobs to a queue using the property name mapred.job.queue.name in the job configuration. There could be a separate configuration file for configuring properties of these queues that is managed by the scheduler. Refer to the documentation of the scheduler for information on the same.
mapred.acls.enabled	Boolean, specifying whether checks for queue ACLs and job ACLs are to be done for authorizing users for doing queue operations and job operations.	If true, queue ACLs are checked while submitting and administering jobs and job ACLs are checked for authorizing view and modification of jobs. Queue ACLs are specified using the configuration parameters of the form mapred.queue.queue-name.acl-name, defined below under mapred-queue-acls.xml. Job ACLs are described at Job Authorization
mapred.task.timeout	Map/reduce task多长时间没有返回就认为他是失败的	一般为10分钟
mapred.map.max.attemptes	如果一个map任务失败，最多可以重新调度的次数	一般为4
mapred.reduce.max.attempts	如果一个reduce任务失败，最多可以重新调度的次数	一般为4
mapred.max.map.failures.percent	一个作业可以允许的map任务失败的比率	有时候我们认为，一个作业即使有一部分任务失败但作业其他任务执行也是有用的，该参数设置一个作业可以允许最多可以承受多少map任务失败
mapred.max.reduce.failures.percent	同上，针对reduce任务	同上，针对reduce任务
mapred.tasktracker.expiry.interval	Tasktracker多长时间没有向JobTracker发送心跳就认为tasktracker失败	默认为10分钟
mapred.user.jobconf.limit	Hadoop能够接受的最多job数量
mapred.tasktracker.map.tasks.maximum	每一个tasktracker能够同时执行的map的个数	默认为2
mapred.tasktracker.reduce.tasks.maximum	每一个tasktracker能够同时执行的reduce的个数	默认为2

下面是Hadoop项目在做一些项目时，对他们的集群所做的配置，在实际工作过程中我们也可以进行参考。

This section lists some non-default configuration parameters which have been used to run the sort benchmark on very large clusters.

l Some non-default configuration values used to run sort900, that is 9TB of data sorted on a cluster with 900 nodes:

Configuration File	Parameter	Value	Notes
conf/hdfs-site.xml	dfs.block.size	134217728	HDFS blocksize of 128MB for large file-systems.
conf/hdfs-site.xml	dfs.namenode.handler.count	40	More NameNode server threads to handle RPCs from large number of DataNodes.
conf/mapred-site.xml	mapred.reduce.parallel.copies	20	Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.
conf/mapred-site.xml	mapred.map.child.java.opts	-Xmx512M	Larger heap-size for child jvms of maps.
conf/mapred-site.xml	mapred.reduce.child.java.opts	-Xmx512M	Larger heap-size for child jvms of reduces.
conf/core-site.xml	fs.inmemory.size.mb	200	Larger amount of memory allocated for the in-memory file-system used to merge map-outputs at the reduces.
conf/core-site.xml	io.sort.factor	100	More streams merged at once while sorting files.
conf/core-site.xml	io.sort.mb	200	Higher memory-limit while sorting data.
conf/core-site.xml	io.file.buffer.size	131072	Size of read/write buffer used in SequenceFiles.

l Updates to some configuration values to run sort1400 and sort2000, that is 14TB of data sorted on 1400 nodes and 20TB of data sorted on 2000 nodes:

Configuration File	Parameter	Value	Notes
conf/mapred-site.xml	mapred.job.tracker.handler.count	60	More JobTracker server threads to handle RPCs from large number of TaskTrackers.
conf/mapred-site.xml	mapred.reduce.parallel.copies	50
conf/mapred-site.xml	tasktracker.http.threads	50	More worker threads for the TaskTracker's http server. The http server is used by reduces to fetch intermediate map-outputs.
conf/mapred-site.xml	mapred.map.child.java.opts	-Xmx512M	Larger heap-size for child jvms of maps.
conf/mapred-site.xml	mapred.reduce.child.java.opts	-Xmx1024M	Larger heap-size for child jvms of reduces

知识点

相关文章

最近更新

Hadoop常用配置总结

相关问答

在docker中搭建hadoop集群有什么好方法配置ip[2024-01-02]

hadoop配置出错问题[2022-11-13]

Nginx是怎么处理不同的端口，常用的配置什么？[2023-04-17]

hadoop配置[2022-10-16]

配置hadoop集群是怎么配置的[2023-12-15]

ubuntu下配置hadoop不能正常运行。。求帮忙[2023-01-29]

dhcp 常用配置项是什么[2024-03-19]

hadoop的最佳配置？(Optimal cofiguration for hadoop?)[2022-03-06]

Hadoop配置属性(Hadoop configuration properties)[2024-01-04]

配置hadoop-client连接到其他机器/服务器中的hadoop(Configure hadoop-client to connect to hadoop in other machine/server)[2021-11-20]