弹性风暴拓扑/ Storm-Hadoop共存(Elastic Storm Topology / Storm-Hadoop Coexisting)

 我们正在评估追求风暴部署，但我有点担心。 我们目前运行Hadoop MapReduce，并希望将我们的一些处理从MapReduce转换到Storm流程。 请注意，这是一些，但不是全部。 我们仍然会有一些MapReduce功能。 
 
 我发现了Mesos，它可能（可能）允许我们在同一硬件上维护Storm和Hadoop部署，但还有其他一些问题： 
 
 
 
 我认为理想的情况是能够任意地“借用”Storm和Hadoop之间的插槽。 恩。 两者都会根据需要使用相同的资源。 不幸的是，这是一个固定的部署，而不是像EC2那样的“基于云”。 
 
 
 我想避免Storm环境中的瓶颈。 理想的情况是根据需求“旋转”（或反转）更多的Bolts实例。 这是可能的/现实的吗？ 
 
 
 “重新启动”拓扑结构似乎是一个相当昂贵的操作，我不确定是否真的是一种选择。 理想情况下，我希望它尽可能无缝。 
 
 
 我们是否正确地处理了这个问题？ 从本质上讲，Storm拓扑将“馈送”MapReduce批处理作业。 我们的一些处理可以以流媒体的方式进行处理，并且会比Storm拓扑好得多，而其中一些需要批处理。 
 
 任何一般的反馈，即使它没有解决我的具体问题，都会受到欢迎。 在这一点上，这更像是一个探索阶段，而我可能完全以错误的方式接近了这一点。 

We are evaluating pursuing Storm for a deployment, but I am a little concerned. We currently run Hadoop MapReduce, and would want to transition some of our processing from MapReduce to Storm processes. Note that that is some, but not all. We would still have some MapReduce functionality.
 
I had found Mesos, which could (potentially) allow for us to maintain a Storm and Hadoop deployment on the same hardware, but had a few other issues:
 
 
 
I envision the ideal situation as being able to "borrow" slots between Storm and Hadoop arbitrarily. ex. both would use the same resources as needed. Unfortunately this is a fixed deployment, and isn't "cloud based" like EC2 or the such.
 
 
I want to avoid bottlenecks in our Storm environment. An ideal case would be to "spin up" (or the inverse) more instances of Bolts as demand requires. Is this possible / realistic?
 
 
"Restarting" a topology seems like a fairly expensive operation, and I'm not sure is really an option. Ideally, I would want it to be as seamless as possible.
 
 
Are we approaching this problem correctly? Essentially, a Storm topology would "feed" a MapReduce batch job. Some of our processing can be processed in a streaming fashion, and would be much better as a Storm topology, while some of it requires batch processing.
 
Any general feedback, even if it doesn't address my specific questions, would be welcome. This is more of an exploratory phase at this point, and I might be totally approaching this the wrong way.

原文：https://stackoverflow.com/questions/14132871