首页 \ 问答 \ 太多的小文件HDFS Sink Flume(Too many small files HDFS Sink Flume)

太多的小文件HDFS Sink Flume(Too many small files HDFS Sink Flume)

agent.sinks=hpd
agent.sinks.hpd.type=hdfs
agent.sinks.hpd.channel=memoryChannel
agent.sinks.hpd.hdfs.path=hdfs://master:9000/user/hduser/gde
agent.sinks.hpd.hdfs.fileType=DataStream
agent.sinks.hpd.hdfs.writeFormat=Text
agent.sinks.hpd.hdfs.rollSize=0
agent.sinks.hpd.hdfs.batchSize=1000
agent.sinks.hpd.hdfs.fileSuffix=.i  
agent.sinks.hpd.hdfs.rollCount=1000
agent.sinks.hpd.hdfs.rollInterval=0
 
 我正在尝试使用HDFS Sink将事件写入HDFS。 并尝试过大小，计数和时间基础滚动，但没有一个按预期工作。 它在HDFS中生成太多小文件，如：  
-rw-r--r--   2 hduser supergroup      11617 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832879.i
-rw-r--r--   2 hduser supergroup       1381 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832880.i
-rw-r--r--   2 hduser supergroup        553 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832881.i
-rw-r--r--   2 hduser supergroup       2212 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832882.i
-rw-r--r--   2 hduser supergroup       1379 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832883.i
-rw-r--r--   2 hduser supergroup       2762 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832884.i.tmp
 
 请协助解决给定的问题。 我正在使用水槽1.6.0  
 〜谢谢 

agent.sinks=hpd
agent.sinks.hpd.type=hdfs
agent.sinks.hpd.channel=memoryChannel
agent.sinks.hpd.hdfs.path=hdfs://master:9000/user/hduser/gde
agent.sinks.hpd.hdfs.fileType=DataStream
agent.sinks.hpd.hdfs.writeFormat=Text
agent.sinks.hpd.hdfs.rollSize=0
agent.sinks.hpd.hdfs.batchSize=1000
agent.sinks.hpd.hdfs.fileSuffix=.i  
agent.sinks.hpd.hdfs.rollCount=1000
agent.sinks.hpd.hdfs.rollInterval=0
 
I'm trying to use HDFS Sink to write events to HDFS. And have tried Size, Count and Time bases rolling but none is working as expected. It is generating too many small files in HDFS like: 
-rw-r--r--   2 hduser supergroup      11617 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832879.i
-rw-r--r--   2 hduser supergroup       1381 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832880.i
-rw-r--r--   2 hduser supergroup        553 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832881.i
-rw-r--r--   2 hduser supergroup       2212 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832882.i
-rw-r--r--   2 hduser supergroup       1379 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832883.i
-rw-r--r--   2 hduser supergroup       2762 2016-03-05 19:37 hdfs://master:9000/user/hduser/gde/FlumeData.1457186832884.i.tmp
 
Please assist to resolve the given problem. I'm using flume 1.6.0 
~Thanks

原文：https://stackoverflow.com/questions/35815163

更新时间：2023-05-12 18:05

最满意答案

 Catalyst::Action::Renderview在决定是否调用视图时使用一Catalyst::Action::Renderview标准。 如果出现以下情况，它将跳过视图：  
 
  该请求是HEAD请求。  
  有些东西已经将$c->response->body设置为定义的值。  
  $c->response->status设置为204（“No Content”）或任何3xx（重定向）。  
  $c->error包含一个或多个错误并且$c->stash->{template}尚未设置（以便finalize_error可以完成其工作）。  
 
 老实说这不是最好的安排，但我会在你的情况下尝试设置$c->res->body(""); 在你的time_server动作中。 一个空体不会写任何东西，并且你的标题已经完成，因为你已经调用了write_fh ，但是仍然定义了一个空字符串，所以它会让RenderView不做任何事情。 

Catalyst::Action::Renderview has a small set of criteria it uses when deciding whether or not to call the view. It will skip running the view if: 
 
 The request was a HEAD request. 
 Something has already set $c->response->body to a defined value. 
 $c->response->status is set to 204 ("No Content") or any 3xx (redirection). 
 $c->error contains one or more errors and $c->stash->{template} hasn't been set (so that finalize_error can do its job instead). 
 
Honestly this isn't the best possible arrangement, but what I would try in your situation is setting $c->res->body(""); in your time_server action. An empty body won't write anything, and your headers are already finalized since you've called write_fh, but an empty string is still defined so it'll keep RenderView from doing anything.

太多的小文件HDFS Sink Flume(Too many small files HDFS Sink Flume)

最满意答案

相关问答

IE9失去对本地Catalyst服务器的请求(IE9 loses requests to a local Catalyst server)[2024-02-27]

Perl Catalyst - 无法呈现模板...未找到(Perl Catalyst - Couldn't render template…not found)[2023-04-03]

为什么Catalyst :: Helper :: View :: TTSite已被弃用？(why is Catalyst::Helper::View::TTSite deprecated?)[2023-09-05]

使用Catalyst和Catalyst :: Plugin :: Email在电子邮件中发送html(Send html in email with Catalyst and Catalyst::Plugin::Email)[2023-08-12]

如何自定义Catalyst :: View :: TTSite？(How can I customize Catalyst::View::TTSite?)[2022-01-14]

Catalyst - 如何跳过渲染视图(Catalyst - How to skip rendering a view)[2023-11-28]

在Catalyst中显式调用视图(Explicit calling a view in Catalyst)[2022-07-21]

Catalyst Controller问题(Catalyst Controller Questions)[2023-09-15]

跳过grails中的视图渲染(Skip the rendering of view in grails)[2023-01-08]

我如何使用Haml和Catalyst？(How can I use Haml with Catalyst?)[2022-10-12]

相关文章

最新问答